Recording medium, reproduction device, program, reproduction method

ABSTRACT

A BD-ROM contains PlayList information. The PlayList information defines a reproduction section for each of a plurality of AV Clips and contains MainPath information and SubPath information. The MainPath information specifies one of the AV Clips as the MainClip and defines a main reproduction section for the MainClip. The SubPath information specifies another one of the AV Clips as the SubClip and defines a sub-reproduction section to be synchronized with the main reproduction section for the SubClip. The BD-ROM contains those AV Clips specifies as SubClips while being correlated with EP_map. The EP_map correlates and indicates the entry time in the time axis of the SubClip with a plurality of entry positions in the SubClip.

TECHNICAL FIELD

The present invention relates to the field of synchronous applicationtechnology.

BACKGROUND ART

Synchronous application technology defines synchronous playback of aplurality of digital streams stored on different recording media, sothat it appears to users to be playback of a single movie.

Synchronous applications are composed of different types of digitalstreams including main streams and substreams. A main stream is adigital stream containing high-quality video, whereas a substream is adigital stream without such high-quality video. Main streams aredistributed to users via large-volume optical discs typified by BD-ROMsstoring the main streams. On the other hand, substreams are distributedto users via, for example, the Internet.

Since digital streams constituting a single movie can be distributed tousers via different media, more flexibility is offered in production ofvariations of the movie. Thus, wider variations can be provided toplayback of a single movie.

Note that the following patent literature discloses prior art relatingto synchronous application technology. Patent Literature 1:

JP Patent Application Publication No. 2002-247526

DISCLOSURE OF THE INVENTION Problems the Present Invention is Going toSolve

In order to ensure playback by consumer home appliances, a synchronousapplication needs to have a mechanism for allowing a consumer homeappliance to execute trick play. The term “trick play” refers to suchplayback operations as fast-forwarding, rewinding, chapter-search, andtime-search. Those playback operations are implemented using the “randomaccess” function to a digital stream. With the random access function,an access is made to a point on the digital stream corresponding to anydesired point on the timeline of the digital stream. In the case of asynchronous application, random access to the main stream is notsufficient to execute trick play. It is also necessary that thesubstream can be accessed at random.

Substreams may carry various kinds of playback data, such as audio,graphics, and standard-quality video. Generally, a main streamcontaining high-quality video is composed of GOPs (Group of Pictures),which are independently decodable units. However, a substream may or maynot be composed of data in any units, such as GOPs. Even if a substreamis composed of data in some units, which are independently decodable,the display rate, sampling frequency, and bit rate of such units maygreatly vary from stream to stream.

Since substreams differ from one another in display rate, samplingfrequency, bit rate, and independently decodable units, random access tosome substream may not be made as fast as random access to the mainstream. Thus, playback of a substream may not be started immediatelyupon request and may be notably delayed.

A synchronous application is designed for synchronous playback of a mainstream and a substream. Thus, delay in playback start of the substreamcauses playback start of the main stream to be delayed as well. As aresult, there is a considerable delay in response to a user request forplayback.

The present invention aims to provide a recording medium and a playbackdevice for avoiding a response delay occurring when random access to asubstream is made in synchronism with random access to the main stream.

Means to Solve the Problems

In an attempt to achieve the above aim, the present invention provides arecording medium having playlist information recorded thereon. Theplaylist information defines a playback section of each of a pluralityof digital streams, and includes main-path information and sub-pathinformation. The main-path information designates one of the digitalstreams as a main stream, and defines a portion of the main stream as aprimary playback section. The sub-path information designates anotherone of the digital streams as a substream, and defines a portion of thesubstream as a secondary playback section that is to be synchronouslyplayed back with the primary playback section. The recording mediumfurther has recorded thereon the one of the plurality of digital streamsdesignated as the substream, together with an entry map. The entry mapindicates a plurality of entry points on the substream in one-to-onecorrespondence with a plurality of entry times on a timeline of thesubstream.

Effects of the Invention

Since an entry map is provided for a digital stream which is asubstream, random access to any desired point on the substream isensured at high speed using the entry map.

The PlayList information defines synchronization of the main stream andthe substream on precondition that high-speed random access is possiblefor both the streams. This arrangement allows a playback device toexecute skip playback and fast-speed playback of a synchronousapplication, which requires synchronous playback of the two streams,immediately upon a user operation.

This advantage will lead to the widespread use of applications made ofmain streams and substreams in combination.

Since high-speed random access to a substream not containing GOPs isensured, playback of a movie composed of a main stream and such asubstream can be started promptly from any point on the timeline. Thatis, a movie composed of a main stream and a substream can be processedin the same manner as a movie composed solely of a main stream.Consequently, a wider and richer variety is offer to playback of asingle movie through a combination of a main stream with a substream.

Here, it should be noted that entry maps provided for substreamsnaturally differ from one another in time accuracy.

Since the time accuracy of each entry map differs depending on the datacarried by a substream associated with the entry map, no information isavailable regarding the time interval between adjacent entry times andwhat kind of data points are designated as entry times. With this beinga situation, it is unknown to what extent the stream analysis needs tobe performed. Without information regarding the extent of requiredstream analysis, it cannot be ensured that random access to thesubstream is made as fast as random access to the main stream. In otherwords, there is an undesirable possibility that playback of thesubstream cannot be started timely and will be delayed notably.

To eliminate the above undesirable possibility, the entry map may be ofa first type or a second type. The first type entry map indicates theplurality of entry times at a constant time interval on the timeline ofthe substream or the plurality of entry points at a constant datainterval on the substream. The second type entry map indicates the entrypoints each located at a head of a complete data set. The entry mapincludes a flag indicating whether the entry map is of the first type orthe second type.

According to the above structure, in the case where the flag indicatesthe first type, the entry points exist either at constant time intervalsor at constant data intervals. Thus, with reference to the flag, theplayback device is informed that stream analysis, if required, needs tocover at most the data worth the constant time interval or the constantdata interval. That is, the playback device is informed that therequested access point is specified by analyzing, even in the worst-caseconditions, data that is worth the constant time interval or theconstant data interval.

In the case where the flag indicates the second type, the entry pointsare located at the heads of complete data sets. Thus, with reference tothe flag, the playback device is informed that data read shall bestarted from any of the entry points, which are located at inconstantintervals. As a result, it is ensured that playback is duly started fromany desired playback point.

Since the flag causes the playback device to judge whether streamanalysis of data worth at most a predetermined time or data interval isrequired or no stream analysis is required at all. Thus, even if notonly the Main Clip but also the SubClip needs to be accessed at random,the playback device is not subject to excessive burden. As a result ofthe reduced burden, response of the playback device to a user operationimproves.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a form of using a recording medium according to thepresent invention;

FIG. 2 is illustrates the internal structure of a BD-ROM;

FIG. 3 schematically illustrates the structure of a file with theextension “.m2ts”;

FIG. 4 illustrates the processes through which TS packets constituting aMain Clip are recorded onto the BD-ROM;

FIG. 5 illustrates the internal structure of a video stream used topresent a movie;

FIG. 6A illustrates the internal structure of an IDR picture,

FIG. 6B illustrates the internal structure of a Non-IDR I picture, andFIG. 6C illustrates the dependency relationship between the Non-IDR Ipicture and other pictures;

FIG. 7 illustrates the processes through which an IDR picture and aNon-IDR I picture are converted into TS packets;

FIG. 8 illustrates the internal structure of Clip information;

FIG. 9 illustrates the setting of an EP_map for a video stream carryinga movie (application_type =1);

FIG. 10 shows pairs of EP_Low and EP_High values representing thePTS_EP_start and the SPN_EP_start of entry points #1-#7 illustrated inFIG. 9;

FIG. 11 illustrates the internal structure of a local storage;

FIG. 12 illustrates the internal structures of a primary audio streamand a secondary audio stream;

FIG. 13 illustrates the internal structure of a PG stream;

FIG. 14 illustrates the internal structure of an IG stream;

FIG. 15 illustrates the data structure of Clip information stored on thelocal storage;

FIG. 16 illustrates an EP_map for a primary or secondary audio stream;

FIG. 17 illustrates an EP_map set on the PG stream timeline;

FIG. 18 illustrates an EP_map set on the IG stream timeline;

FIG. 19 illustrates the data structure of PlayList information;

FIG. 20 illustrates the relationship between the Main Clip and thePlayList information;

FIG. 21 illustrates the internal structure of PlayListMark informationin PlayList information;

FIG. 22 illustrates chapter positions specified by the PLMarkinformation included in the PlayList information;

FIG. 23 illustrates the internal structure of SubPath information indetail;

FIG. 24 illustrates the relationship among the Main Clip stored on theBD-ROM and the SubClip and the PlayList information both stored on thelocal storage;

FIG. 25 illustrates the EP_map for the Main Clip, a PlayItem timeline,the EP_map for the SubClip, which is either a primary or secondary audiostream, and a SubPlayItem timeline;

FIG. 26 illustrates the EP_map for the Main Clip, the PlayItem timeline,the EP_map for the SubClip, which is either a PG or IG stream;

FIG. 27 shows a table of permissible combinations of the values of theSubPath_type, the application_type, and the EP_stream_type;

FIG. 28 illustrates the virtual file system created by a playback device300;

FIG. 29 illustrates the internal structure of the playback deviceaccording to the present invention;

FIG. 30 is a flowchart illustrating the processing steps for executingjump playback;

FIG. 31 schematically illustrate how a random access point is determinedusing the EP_map illustrated in FIG. 25;

FIG. 32 is a flowchart illustrating the processing steps for convertinga coordinate TM on the Main Clip and the SubClip to a correspondingaddress;

FIG. 33 illustrates the relationship among variables k and h and therandom access point in the case where the SubClip is a primary orsecondary audio stream;

FIG. 34 illustrates the relationship among the variables k and h and therandom access point in the case where the SubClip is a PG or IG stream;

FIG. 35 illustrates one example of PiP playback;

FIG. 36A illustrates an HD image and an SD image for comparison, and

FIG. 36B illustrates how the secondary video is scaled up and down;

FIG. 37 illustrates the contents stored on the local storage accordingto a second embodiment;

FIG. 38 illustrates the internal structure of clip information stored onthe local storage according to the second embodiment;

FIG. 39 illustrates an EP_map for the secondary video stream in the samemanner to FIG. 9;

FIG. 40 illustrates PlayList information defining a synchronousapplication implementing statically synchronized PiP playback;

FIG. 41 illustrates, in the same manner as FIGS. 25 and 26, how thePlayList information defines the synchronization between the Main Clipcontaining primary video and the SubClip containing secondary video;

FIGS. 42A-42C illustrate a dynamically synchronized PiP application;

FIG. 43 illustrates the internal structure of the PlayList informationdefining dynamically synchronized PiP playback;

FIG. 44 illustrates the internal structure of a playback deviceaccording to the second embodiment;

FIG. 45 is a flowchart illustrating the processing steps for executingPL playback;

FIG. 46 schematically illustrates, in the same manner as FIG. 31, randomto the Main Clip and to the SubClip; and

FIG. 47A illustrates the playback control for executing dynamicallysynchronized PiP playback,

FIG. 47B illustrates dynamically synchronized PiP playback executed inthe case where a rock point once passed is reached again in a normalplayback mode as a result of rewinding, and FIG. 47C illustrates PiPplayback executed in the case where a playback section of the secondaryvideo is located at some point subsequent to the end point of a playbacksection of the primary video.

DESCRIPTION OF REFERENCE NUMERALS

1 BD Drive

2 Arrival Time Clock Counter

3 Source De-Packetizer

4 PID Filter

5 Transport Buffer

6 Multiplexed Buffer.

7 Coded Picture Buffer

8 Video Decoder

10 Decoded Picture Buffer

11 Video Plane

12 Transport Buffer

13 Coded Data Buffer

14 Stream Graphics Processor

15 Object Buffer

16 Composition Buffer

17 Composition Controller

18 Presentation Graphics Plane

19 CLUT Unit

20 Transport Buffer

21 Coded Data Buffer

22 Stream Graphics Processor

23 Object Buffer

24 Composition Buffer

25 Composition Controller

26 Interactive Graphics Plane

27 CLUT Unit

28 Compositor

29 Compositor

30 Switch

31 Network Device

32 Switch

33 Arrival Time Clock Counter

34 Source De-Packetizer

35 PID Filter

36 Switch

37 Transport Buffer

38 Elementary Buffer

39 Audio Decoder

40 Transport Buffer

41 Buffer

42 Audio Decoder

43 Mixer

44 Scenario Memory

45 Controller

46 Main Exchanger

47 Sub-Exchanger

48 PL Playback Controller

49 PSR Set

BEST MODE FOR CARRYING OUT THE INVENTION First Embodiment

The following describes an embodiment of a recording medium according tothe present invention. First of all, among various acts of practicing arecording medium of the present invention, an act of using is described.FIG. 1 illustrates a form of using the recording medium according to thepresent invention. In FIG. 1, a local storage 200 is the recordingmedium according to the present invention. The local storage 200 is ahard disk built into a playback device 300. The local storage 200 isused with a BD-ROM 100 to together supply a movie to a home theatersystem composed of the playback device 300, a remote controller 400, anda television set 500.

The local storage 200 is a built-in hard disk used to store contentssupplied from movie distributors.

The playback device 300 is a network-capable digital home appliance andused to play back the BD-ROM 100. In addition, the playback device 300can expand the functionality of the BD-ROM 100 through the combined useof the contents stored on the BD-ROM 100 with contents downloaded from amovie distributor's server via the network.

The remote controller 400 is used to receive user input specifying, forexample, a chapter to be played back or time to start playback.

The television set 500 displays playback produced by the playback device300.

The contents stored on the local storage 200 are used in combinationwith the contents stored on the BD-ROM 100. Here, the contents stored onthe local storage 200 and not physically present on the BD-ROM 100 areprocessed as if they are stored on the BD-ROM. This technique is called“virtual file system”.

The recording medium according to the present invention is designed forsuch a use in combination with the BD-ROM 100. This concludes thedescription of the act of using the recording medium of the presentinvention.

Now, the following sequentially describes the internal structures of theBD-ROM 100 and of the local storage 200.

<Overview of BD-ROM>

FIG. 2 illustrates the internal structure of the BD-ROM. In the figure,the BD-ROM is illustrated on the fourth level, and the track of theBD-ROM is illustrated on the third level. In the figure, the track islaterally stretched out, although the track in practice spirals outwardsfrom the center of the BD-ROM. The track is composed of a lead-in area,a volume area, and a lead-out area. The volume area has a layer model ofa physical layer, a file system layer, and an application layer. Thefirst level illustrates, in a directory structure, a format of theapplication layer (application format) of the BD-ROM. As illustrated onthe first level, the BD-ROM has a ROOT directory, and the ROOT directoryhas a BDMV directory.

The BDMV directory has three subdirectories called a PLAYLIST directory,a CLIPINF directory, and a STREAM directory.

The STREAM directory stores files with the extension “.m2ts” (e.g. afile called 00001.m2ts) containing data constituting the main body of adigital stream.

The PLAYLIST subdirectory stores files with the extension “.mpls” (e.g.a file called 00001.mpls).

The CLIPINF directory stores files with the extension “.clpi” (e.g. afile called 00001.clpi).

The following describes an AV clip, Clip Information, and PlayListinformation, which are components of the BD-ROM.

<BD-ROM Structure: 1. AV Clip>

First of all, a description is given of files with the extension“.m2ts”. FIG. 3 schematically illustrates the structure of a file withthe extension “.m2ts”. Each file with the extension “.m2ts” (namely,00001.m2ts, 00002.m2ts, 00003.m2ts, ) contains an AV clip. The AV clip(illustrated on the fourth level) is created as follows. A video streamcontaining a plurality of video frames (pictures pj1, pj2, pj3, . . . )and an audio stream containing a plurality of audio frames (bothillustrated on the first level) are separately converted to PES packets(illustrated on the second level) and further converted to TS packets(illustrated on the third level). These TS packets of the video andaudio streams are multiplexed to form the AV clip. As in the exampleillustrated in FIG. 3, an AV clip containing video is specificallyreferred to as a “Main Clip” to differentiate it from other AV clips notcontaining video.

Next, a description is given of how a Main Clip, which is a digitalstream of the MPEG2-TS format, is recorded on the BD-ROM. FIG. 4illustrates the processes through which TS packets constituting the MainClip are recorded onto the BD-ROM. In FIG. 4, the TS packets of the MainClip are illustrated on the first level.

Each TS packet of the Main Clip is 188 bytes in length. As illustratedon the second level, “TS_extra_header” (denoted as “EX” in the figure)is attached to each TS packet, so that the packet length becomes 192bytes.

The third and fourth levels illustrate the physical units of the BD-ROMin relation to the TS packets. As illustrated on the fourth level, thetrack on the BD-ROM is divided into a plurality of sectors. The TSpackets with the TS_extra_header (hereinafter, simply “EX-TS packets”)are divided into groups of 32 EX-TS packets, and each group is writteninto three sectors of the BD-ROM. Each group composed of 32 EX-TSpackets amounts to 6,144 (=32×192) bytes, which is equal to the totalsize of three sectors (=2048×3) Each group of 32 EX-TS packets stored inthree sectors of the BD-ROM is referred to as an “Aligned Unit”. Whendata is recorded onto the BD-ROM, encryption of data is carried out inAligned Units.

As illustrated on the fifth level, an error correction code is insertedevery 32 sectors to constitute an ECC block. As long as accessing theBD-ROM in Aligned Units, a playback device is ensured to obtain acomplete set of 32 EX-TS packets without missing any part. Thiscompletes the description of the processes of recording the Main Cliponto the BD-ROM.

<Description of AV Clip: 1. Video Stream>

The following describes the internal structure of a video stream.

FIG. 5 illustrates the internal structure of a video stream used topresent a movie. The video stream illustrated in FIG. 5 is composed of aplurality of pictures aligned in the order of coding.

In the figure, the reference numerals “I”, “P”, and “B” denote an Ipicture, B picture, and P picture, respectively. There are two types ofI pictures, one is an IDR picture and the other is a Non-IDR I picture.Non-IDR I pictures, P pictures, and B pictures are coded usingcorrelation with other pictures. More specifically, a B picture iscomposed of Bidirectioanlly Predictive (B) slices. A P pictureis_composed of Predictive (P) slices. There are two types of B pictures,one is a reference B picture, and the other is a nonreference B picture.

In FIG. 5, a Non-IDR I picture is denoted as “I”, and an IDR picture isdenoted as “IDR”. The same denotations are used throughout the followingdescriptions. This concludes the description of the video stream used topresent a movie.

Next, the following describes the internal structures of an IDR pictureand a Non-IDR I picture. FIG. 6A illustrates the internal structure ofan IDR picture. As illustrated in the figure, the IDR picture iscomposed of a plurality of Intra-type slices. FIG. 6B illustrates theinternal structure of a Non-IDR I picture. Different from the IDRpicture composed solely of Intra-type slices, the Non-IDR I picture iscomposed of slices of Intra-type, P-type, and B-type. FIG. 6Cillustrates the dependency relationship between the Non-IDR I pictureand other pictures. A Non-IDR I picture may be composed of B and Pslices and thus may have a dependency relationship with other pictures.

<Description of AV Clip: 2. Recording to BD-ROM>

Next, the following describes how IDR pictures and Non-IDR I picturesare converted into TS packets and recorded onto the BD-ROM. FIG. 7illustrates the processes through which an IDR or Non-IDR I picture isconverted into TS packets. In the figure, the first level illustrates anIDR or Non-IDR I picture. The second level illustrates an Access Unitstipulated according to MPEG4-AVC. A plurality of slices constitutingthe IDR or Non-IDR I picture is sequentially aligned. Then, AUD (AccessUnit Delimiter), SPS (Sequence Parameter Set), PPS (Picture ParameterSet), SEI (Supplemental Enhanced Information) are attached to the slicesequence. In this way, the picture slices are converted into an AccessUnit.

AUD, SPS, PPS, SEI, Access Unit mentioned above are information allstipulated according to MPEG4-AVC and described in various documents,such as “ITU-T Recommendation H.264”. For the details, such documentsshould be referenced. The point in this description is that AUD, SPS,PPS, and SEI need to be supplied to a playback device for allowingrandom access to the video stream.

The third level represents NAL units. AUD, SPS, PPS, SEI, and each sliceillustrated on the second level are each attached with a header so as tobe converted into separate NAL units. NAL units are units supported bythe Network Abstraction Layer (NAL) stipulated according to MPEG-4AVCand described in various documents, such as “ITU-T RecommendationH.264”. For the details, such documents should be referenced. The pointin this description is that AUD, SPS, PPS, SED, and each slice areconverted into separate NAL units and manipulated independently in theNetwork Abstraction Layer.

As described above, a plurality of NAL units obtained by converting thesingle picture are further converted into PES packets illustrated on thefourth level. The PES packets are then converted into TS packets.Finally, the resulting TS packets are recorded onto the BD-ROM.

In order to play back one GOP, the decoder needs to be supplied with aNAL unit containing AUD, among NAL units constituting the first IDR orNon-IDR I picture in the GOP. That is, the NAL unit containing AUD isused as an index for decoding the IDR picture or Non-IDR I picture. Inthis embodiment, each NAL unit containing AUD is regarded as a point.For playback of the video stream, the playback device recognizes each ofsuch a point as an entry point for playback of a Non-IDR I picture andan IDR picture. Consequently, for implementation of random access to theMain Clip, it is extremely important for the playback device torecognize the locations of AUDs of IDR pictures and Non-IDR I pictures.This concludes the description of the structure of the MPEG-4AVC videostream used for presenting a movie.

<BD-ROM Structure: 2. Clip Information>

Next, the following describes files with the extension “.clpi”. Eachfile with the extension “.clpi” (e.g. 00001.clpi, 00002.clpi,00003.clpi, . . . ) contains Clip information. Each piece of Clipinformation is management information of an individual Main Clip. FIG. 8illustrates the internal structure of a piece of Clip information. Asillustrated in the left block of the figure, the Clip information iscomposed of the following fields:

(i) “ClipInfo( )” storing the attributes of the AV clip file;

(ii) “Sequence Info( )” storing information related to the ATC sequenceand STC Sequence;

(iii) “Program Info( )” storing information related to the programsequence; and

(iv) “Characteristics Point Info (i.e., CPI ( ))”.

<Description of Clip information: 1. ClipInfo>

First, a description of ClipInfo will be given. In the figure, leaderlines ct1 indicates that the structure of ClipInfo( ) is illustrated ingreater detail. As indicated by the leader lines ct1, the ClipInfo( ) iscomposed of the following fields: “clip_stream_type” indicating the typeof the digital stream; “application_type” indicating the type of anapplication that the Main Clip constitutes; and “TS_recording rate”indicating the data rate of the Main Clip. The application_type is setto the value “1” to indicate that the associated Main Clip constitutes aMovie application. This concludes the description of ClipInfo( ).

Next, the following describes ClipInfo.

The ClipInfo includes as many “EP_map_for_one_stream[0]—[Ne−1]” fieldsas the value held by “Ne”, and also includes the following attributes ofeach “EP_map_for_one_stream”. The attributes of each“EP_map_for_one_stream” include: “stream_PID[0]—[Ne−1]” of an associatedelementary stream; “EP_stream_type[0]—[Ne−1]” indicating the type of theEP_map_for_one_stream; “number_of_EP_High entries[0]—[Ne−1]” indicatingthe number of EP_High entries in the EP map_for_one_stream;“number_of_EP_Low_entries[0]—[Ne−1]” indicating the number of EP_Lowentries in the EP_map_for_one_stream; and“EP_map_for_one_stream_PID_start_address[0]—[Ne−1]” indicating thecurrent address of the EP_map_for_one_stream.

<Clip information: 2. EP_map>

Hereinafter, the EP_map is described using a specific example. FIG. 9illustrates the setting of an EP_map for a video stream carrying a movie(application_type=1). The first level illustrates a plurality ofpictures aligned in the presentation order. The second level illustratesthe timeline of the pictures. The fourth level illustrates TS packets onthe BD-ROM. The third level illustrates the EP_map setting.

In this example, Access Units (Non-IDR I pictures and IDR pictures) arelocated at points t1-t7 on the timeline illustrated on the second level.Provided that the time intervals between the points t1-t7 are about 1sec, the EP_map of the movie video stream is set so as to indicate eachof the points t1-t7 with a corresponding entry time (PTS_EP_start) and acorresponding entry point (SPN_EP_start).

FIG. 10 shows, in tabular form, pairs of EP_Low and EP_High valuesrepresenting the PTS_EP_start and the SPN_EP_start of entry points #1-#7illustrated in FIG. 9. In FIG. 10, the left table shows the values ofEP_Low and the right table shows the values of EP_High.

In FIG. 10, of the EP_Low(0)−(Nf−1) shown in the left table, eachPTS_EP_Low value of EP_Low(i)−(i+3) indicate the least significant bitsof the points t1-t4. Also, each SPN_EP_Low value of the EP_Low(i)−(i+3)indicate the least significant bits of the points n1-n4.

The right table in FIG. 10 shows the values of EP_High(0)−(Nc−1) set inthe EP_map. Provided that the points t1-t4 have the same set of mostsignificant bits and the points n1-n4 have the same set of mostsignificant bits, the values of PTS_EP_High and SPN_EP_High are set tothe respective sets of most significant bits. In addition, theref_to_EP_Low_id field corresponding to the EP_High is set, so that thefirst EP_Low field (EP_Low(i)) out of the EP_Low fields indicating thepoints t1-t4 and n1-n4. With this setting, the EP_High indicates thecommon most significant bits of the PTS_EP_start and the common mostsignificant bits of the SPN_EP_start. This concludes the description ofthe Main Clip and Clip information stored on the BD-ROM. Note that the00001.mpls file is a file storing the PlayList information, but nodescription thereof is given here. It is because an identical file isstored on the local storage 200 and a description of the PlayListinformation stored on the local storage 200 will be given later.

Since the BD-ROM is a large-capacity recording medium, it is understoodthat the BD-ROM is used to supply video to the home theater systemillustrated in FIG. 1. This concludes the description of the-BD-ROM.

<Local Storage 200>

Next, the following describes the local storage 200, which is arecording medium according to the present invention. FIG. 11 illustratesthe internal structure of the local storage 200. As illustrated in thefigure, the recording medium according to the present invention can bemanufactured by making improvements to the application layer.

In the figure, the local storage 200 is illustrated on the fourth level,and the track of the local storage 200 is illustrated on the thirdlevel. In the figure, the track is laterally stretched out, although thetrack in practice spirals outwards from the center of the local storage200. The track consists of a lead-in area, a volume area, and a lead-outarea. The volume area in the figure has a layered structure made up of aphysical layer, a file system layer, and an application layer. The firstlevel of the figure illustrates the application format of the localstorage 200 using a directory structure.

In the directory structure illustrated in the figure, the local storage200 has a ROOT directory. The ROOT directory has a subdirectory called“organization #1”, and the “organization #1” has a subdirectory called“disc #1”. The “organization #1” directory is assigned to a specificmovie provider, whereas the “disc #1” directory is assigned to aspecific BD-ROM supplied from that provider.

Since a specific directory is assigned to a specific provider andsubdirectories are provided within the directory in one-to-onecorrespondence with BD-ROMs supplied from that provider, download datarelating to each BD-ROM is stored separately in the respectivesubdirectories. Similarly to the contents stored on the BD-ROM, thesubdirectories contain PlayList information (00002.mpls), Clipinformation (00002.clpi, 00003.clpi, 00004.clpi, and 00005.clpi), and AVclips (00002.m2ts, 00003.m2ts, 00004.m2ts, and 00002.m2ts).

Next, the following describes PlayList information, Clip information,and AV clips, all of which are component of the local storage 200.

<Structure of Local Storage 200: 1. AV Clip>.

The AV clips (00002.m2ts, 00003.m2ts, 00004.m2ts, and 00005.m2ts) storedon the local storage 200 constitutes SubClips each of which is an AVclip composed of one ore more Out-of-MUX streams. An Out-of-MUX streamrefers to an elementary stream to be presented during playback of an AVclip containing a video stream but not multiplexed with the videostream. In addition, a process of reading and decoding an Out-of-MUXstream to playback the Out-of-MUX stream during video stream playback isreferred to as “Out-of-MUX stream frame work”.

The types of Out-of-MUX streams include “primary audio stream”,“secondary audio stream”, “presentation graphics (PG) stream”, and“interactive graphics (IG) stream”.

According to the embodiment 1, among four AV clips illustrated in FIG.11, the 00002.m2ts file contains a primary audio stream, the 00003.m2tsfile contains a secondary audio stream, the 00004.m2ts file contains aPG stream, and the 00005.m2ts file contains an IG stream. Yet, thestored contents are just one example and it is applicable that the fourOut-of-MUX streams are multiplexed into one SubClip. Hereinafter, thedetails of Out-of-MUX streams will be described.

<Description of Out-of-MUX Stream: 1. Primary Stream & Secondary Stream>

The “primary audio stream” carries main audio data, whereas the“secondary audio stream” caries auxiliary audio data. At the time ofSubClip playback, audio playback of the secondary audio stream is mixedwith audio playback of the primary audio stream before output. Examplesof audio data carried by the secondary audio stream include“commentary”. In the case where the primary audio stream provides speechand BGM of a movie, which is a main feature, and the secondary streamprovides director's commentary, the sound of speech and BGM of the movieis mixed with the audio of the commentary before output.

The secondary audio stream is stored only on the local storage 200 andsupplied for playback, but not stored on the BD-ROM. On the other hand,the primary audio stream may be stored on the BD-ROM or on the localstorage 200. In addition, the primary audio stream may be coded using adifferent scheme from that used for coding the secondary audio stream.

FIG. 12 illustrates the internal structure of either of the primary andsecondary audio streams. In the figure, the first level illustrates thetimeline referenced during SubClip playback (SubClip timeline). Thesecond level illustrates TS packets constituting the SubClip. The thirdlevel illustrates PES packets constituting the SubClip. The fourth levelillustrates audio frames constituting the primary or secondary audiostream. As illustrated in the figure, the SubClip is manufactured byconverting frames of the primary or secondary audio stream into PESpackets (illustrated on the third level), and further converting the PESpackets into TS packets (illustrated on the second level).

The header of a PES packet contains PTS indicating the presentationstart time of the audio frame contained in the PES packet. That is, withreference to the PTS, the playback timing of the audio frame containedin the PES packet is specified on the SubClip timeline. Thus, the headerof each PES packet is subjected to the stream analysis.

<Description of Out-of-MUX Stream: 2. PG Stream>

A PG stream is an elementary stream for synchronously presentingsubtitles with video playback. FIG. 13 illustrates the internalstructure of a PG stream. In the figure, the fourth level illustratesPES packets constituting the SubClip, and the third level illustrates TSpackets constituting the SubClip. The second level illustrates theSubClip timeline. The first level illustrates composite picturesobtained by decoding and overlaying the PG stream contained in theSubClip on the video stream contained in the Main Clip.

The following describes the structure of the PES packets constitutingthe SubClip, illustrated on the fourth level.

Each PES packet of the SubClip is generated by attaching a PES packetheader to a sequence of functional segments, namely PCS (PresentationControl Segment), PDS (Pallet Definition Segment), WDS (WindowDefinition Segment), ODS (Object Definition Segment), and END (END ofDisplay Set Segment).

The ODS defines graphics data for presentation of text subtitles.

The WDS defines a rendering area for the display of graphics data on ascreen.

The PDS defines output colors for rendering the graphics data.

The PCS defines page control for presentation of text subtitles. Thepage control includes Cut-In/Out, Fade-In/Out, Color Change, Scroll,Wipe-In/Out. Under the page control defined by PCS, it is possible torealize a display effect of gradually erasing a set of text subtitlesand presenting a next set of text subtitles.

The END defines the end of the sequence of functional segments definingpresentation of text subtitles.

The PES packet header includes timestamps, such as PTS and DTS. Thetimestamps indicate the start timing of decoding the functional segmentsand the presentation timing of the graphics defined by the functionalsegments. The sequence of functional segments starting with the PCS andend with END is referred to as a “Display Set (DS)”. The third levelillustrates TS packets obtained by converting the PES packets. Thesecond level illustrates the timeline referenced for playback of theSubClip (SubClip timeline). The DTS included in the PCS indicates on theSubClip timeline the timing for decoding the PCS. The PTS included inthe PCS indicate on the SubClip timeline the presentation timing ofgraphics defined by the DS, which starts with the PCS. At the timingindicated by the PTS, the composite pictures illustrated on the firstlevel are displayed.

The types of DSs include “Epoch Start”, “Acquisition Point”, “NormalCase”, and “Epoch Continue”.

An “Epoch Start” DS indicates the start of a new Epoch. An Epoch is atime period of continuous memory management on the timeline of the AVclip playback, and also refers to a group of data allocated to that timeperiod. Thus, an Epoch Start DS contains all functional segmentsnecessary for the next screen composition. An Epoch Start DS is providedat a point that will be accessed during skip playback, such as the startof a chapter in the movie.

An “Acquisition Point” DS is not the start of an Epoch but contains allfunctional segments necessary for the next screen composition.Therefore, graphics can be reliably displayed if playback is startedfrom an Acquisition Point DS. That is to say, an Acquisition Point DSenables a display composition to be made from a midpoint in the Epoch.An Acquisition Point DS is provided at a point to which a skip operationmay be made.

A “Normal Case” DS contains only a difference from the immediatelypreceding DS. For example, if the subtitles presented by a DSv areidentical to the subtitles presented by the immediately preceding DSubut different in screen layout, the DSv contains a PCS and an END onlyand serves as a Normal Case DS. This arrangement eliminates the need toprovide overlapping ODSs in DSs, so that the amount of data stored onthe BD-ROM is reduced. Since a Normal Case DS only contains thedifference, no display composition can be presented with the Normal CaseDS alone.

An “Epoch Continue” DS indicates the Epoch of the AV clip continues fromanother AV clip if playback of the two AV clips shall be continuous.This concludes the description of the functional segments constitutingthe PG stream.

<Description of Out-of-MUX Stream: 3. IG Stream>

The IG stream is an elementary stream for synchronous presentation oftext subtitles with video playback.

FIG. 14 illustrates the internal structure of an IG stream. The fourthlevel illustrates PES packets constituting a SubClip. The third levelillustrates TS packets constituting the SubClip. The second levelindicates the SubClip timeline. The first level illustrates compositepictures obtained by decoding and overlaying the IG stream, which is anOut-of-MUX stream, and the video stream, which is the Main Clip.

Now, the following describes the PES packets illustrated on the fourthlevel.

The PES packets of the SubClip is generated by attaching a PES packetheader to a sequence of functional segments, namely ICS (InteractiveComposition Segment), PDS (Pallet Definition Segment), ODS (ObjectDefinition Segment), and END (END of Display Set Segment).

The ODS defines graphics data used for rendering a button.

The PDS defines output colors used for rendering graphics data.

The ICS defines interactive control for changing the button state inresponse to a user operation.

The END defines the end of the sequence of functional segments definingpresentation of a menu.

The header of a PES packet includes timestamps, such as PTS and DTS. Thetimestamps indicate the start timing of decoding the functional segmentsand the presentation timing of the graphics defined by the functionalsegments. The sequence of functional segments starting with the PCS andend with END is referred to as a “Display Set”. Similarly to DSscontained in the PG stream, the types of DSs include “Epoch Start”,“Acquisition Point”, “Normal Case”, and “Epoch Continue”.

In FIG. 14, the third level illustrates TS packets obtained byconverting the PES packets. The second level illustrates the timelinereferenced during SubClip playback (SubClip timeline). The DTS includedin the ICS indicates on the SubClip timeline the timing for decoding theICS. The PTS included in the ICS indicate on the SubClip timeline thepresentation timing of graphics defined by the DS, which starts with theICS. At the timing indicated by the PTS, the composite picturesillustrated on the first level are displayed.

A sequence of functional segments including ICS, PDS, ODS, and END isreferred to as a “Display Set”, which is a group of functional segmentsfor realizing display of one menu.

This concludes the description of SubClip.

<Structure of Local Storage 200: 2. Clip Information>

The following describes Clip information stored on the local storage200. FIG. 15 illustrates the data structure of Clip information storedon the local storage 200. The Clip information stored on the localstorage 200 is identical in data structure to the Clip informationstored on the BD-ROM 200. Yet, the application_type, EP_map, andEP_stream_type fields are set to the values specific to the SubClip.

<Description of Clip Information in SubClip: 1. Application_type>

The following describes the application_type field illustrated in FIG.15. If the SubClip is one of primary audio stream, secondary audiostream, PG stream, and IG stream mentioned above, the applicationtype_is set to the value “7”.

The application_type=7 indicates that the SubClip multiplexed with theClip information is “additional contents without video”. Since no videois contained, the SubClip is designed to be stored to the local storage200 via the network and to be supplied to a playback device from thelocal storage 200 rather than from the BD-ROM. This concludes thedescription of the setting of the application_type field in the SubClip.

<Clip Information in SubClip: 1. Out-of-MUX_EP_map>

Next, the EP_map provided for the SubClip is described. The EP_mapfields on the local storage 200 are presented one for each Out-of-MUXstream multiplexed in the SubClip. Each EP_map indicates a plurality ofentry points of the associated Out-of-MUX stream, along with a pluralityof entry times corresponding to the entry points.

The types of Out-of-MUX streams include “primary audio stream”,“secondary audio stream”, “PG stream”, and “IG stream”. Each type ofOut-of-MUX stream differs in point on the stream at which the streambecomes available for playback. More specifically, the primary andsecondary audio streams are each composed of a plurality of audio framesand playback can basically be started from the head of any audio frame.

In the case of PG and IG streams, on the other hand, it is required thata Display Set composed of a complete set of functional segments besupplied to a decoder. A DS composed of a complete set of functionalsegments refers to a DS other than “Normal Case”. That is, “EpochStart”, “Acquisition Point”, and “Epoch Continue” DSs are each composedof a complete set of functional segments, and the first PCS and thefirst ICS in such DSs serve as entry points.

As described above, each type of Out-of-MUX stream differs in point onthe stream at which decoding of the stream can be started. Thus, eachEP_map has a different structure depending on the type of. Out-of-MUXstream associated with the EP_map. An EP_map associated with anOut-of-MUX stream is referred to as an “Out-of-MUX_EP_map”.

<Details of Out-of-MUX_EP_map: 2. Out-of-MUX_EP_map for Audio>

Next, the following describes the structure of EP_map associated with aprimary or secondary audio stream. The EP_map associated with a primaryor secondary audio stream differs from the EP_map associated with videoin time intervals between adjacent entry points. More specifically, inthe EP map for video, entry points are set at relatively short timeintervals not greater than one second. On the other hand, entry pointsin the EP_map for audio are set at longer time intervals of fiveseconds.

FIG. 16 illustrates the EP_map for a primary or secondary audio stream.The structure of the EP_map illustrated in the figure is such that entrypoints are set so as to correspond to entry times set at constant timeintervals of five seconds. In the figure, the third level illustratesthe SubClip timeline, and t1-t6 on the SubClip timeline denote entrytimes. The entry times t1, t2, t3, t4, t5, and t6 are set at constanttime intervals of five seconds. The second level illustrates the EP_map.Each PTS_EP_start field illustrated on the second level indicates anentry time.

The first level illustrates TS packets constituting the primary orsecondary audio stream. Among the TS packets, SPN=n1, n2, n3, n4, n5,and n6 are set to indicate entry points, and the SPN_EP_start fieldsillustrated on the second level are set to indicate n1-n6. Since theSPN_EP_start of each entry point is set to correspond PTS_EP_start, theentry times at five seconds intervals correspond to entry points.

In the EP_map of the primary or secondary audio stream, the intervalbetween each two adjacent entry points may be the data interval of 256Kbytes. This interval of 256K bytes is calculated by multiplying thetransfer rate of the secondary audio stream by the time interval of fiveseconds. Since the time interval set in the EP map is five seconds, thestream analysis needs to cover at most five seconds of data. Thisconcludes the description of the EP_map set for the primary or secondaryaudio stream.

<Details of Out-of-MUX_EP_map: 3. EP_map for PG Stream>

Next, an EP_map set for a PG stream will be described.

FIG. 17 illustrates the EP_map set on the PG stream timeline. In thefigure, the first level illustrates TS packets constituting the PGstream. The second level illustrates the EP_map. The third levelillustrates the SubClip timeline. On the first level, the first PCS ineach DS which is not a Normal Case DS is present at points specified bySPN=n1 and n5. These points n1 and n5 are determined as entry points. Onthe other hand, the PTSs of the PCSs specify the temporal points t1 andt5 on the SubClip timeline illustrated on the third level. Thus, theEP_map shows SPN=n1 and n5, together with the corresponding PTS=t1 andt5, respectively.

As above, the head of each complete set of functional segments coincideswith an entry time shown by the PTS of PCS. Consequently, with referenceto the EP_map, presentation of graphics can be started from some midpoint of the stream without conducting stream analysis.

<Details of Out-of-MUX_EP_map: 4. EP_map for IG Stream>

FIG. 18 illustrates the EP_map set on the IG stream timeline. In thefigure, the first level illustrates TS packets constituting the IGstream. The second level illustrates the EP_map. The third levelillustrates the SubClip timeline. On the first level, the first ICS ineach DS which is not a Normal Case DS is present at points specified bySPN=n1 and n5. These points n1 and n5 are determined as entry points. Onthe other hand, the PTSs of the ICSs specify the temporal points t1 andt5 on the SubClip timeline illustrated on the third level, the EP_mapshow the SPN=n1 and n5, together with the corresponding PTS=t1 and t5,respectively.

As above, the head of each complete set of functional segments coincideswith an entry time shown by the PTS of ICS. Consequently, with referenceto the EP_map, presentation of menu display can be started from some midpoint of the stream without conducting stream analysis.

The above description clarifies that the structure of EP_map differsdepending on the type of Out-of-MUX stream associated with the EP_map.This concludes the description of the EP_map included in the Clipinformation stored on the local storage 200. The following describes theEP_stream_type field included in the Clip information stored on thelocal storage 200.

<Description of Clip Information in SubClip: 3. EP_stream_type>

The EP_stream_type is provided for each Out-of-MUX stream multiplexed inthe SubClip and indicates one of the EP_map structures illustrated inFIGS. 16-18. That is, when i-th Out-of-MUX stream multiplexed in oneSubClip is denoted as Out-of-MUX stream[i], the EP_stream_type[i]indicates the type of EP_map_for_one_stream associated with theOut-of-MUX stream[i] and set to one of the values of “3”, “4”, “6”, and“7”. When set to the value of “3” or “4”, the EP_stream_type indicatesthe EP_map structure specific to a primary or secondary audio stream.That is, the EP_map of that type shows entry times set at constant timeintervals or entry points set at constant data intervals, as illustratedin FIG. 16. When set to the value of “6”, the EP_stream type indicatesthe EP_map structure specific to a PG stream. That is, the EP_map ofthat type shows, as entry points, the head of each DS other than NormalCase. DS (as illustrated in FIG. 17). When set to the value of “7”, theEP_stream type indicates the EP_map structure specific to the IG stream.That is, the EP_map of that type shows, as entry points, the head ofeach DS other than Normal Case DS (as illustrated in FIG. 18).

Since the EP_stream_type clearly indicates the structure of theassociated EP_map, it is known whether stream analysis is required froman entry point shown by the EP_map or data is to be read starting froman entry point shown by the EP_map and supplied to the decoder.

<Description of Clip Information in SubClip: 4. Significance ofEP_stream_type>

The following describes the technical significance of the provision ofthe EP_stream_type field in the Out-of-MUX_EP_map field.

In the case where an entry time is specified as an access point,playback can be started immediately. However, in the case where arequested access point does not coincide with an entry time, streamanalysis needs to be carried out. If no information is provided as tothe time intervals between adjacent entry times or the types of datadesignated as entry points, the playback device cannot estimate themaximum degrees of stream analysis that would be sufficient. Not knowingthe maximum degree of stream analysis that would be required, it ispossible that a response of the playback device to a user operation willbe notably delayed.

In view of the above undesirable possibility, the Out-of-MUX_EP_mapincludes the EP_stream_type that indicates the time intervals betweenentry times and the data types of entry points. When set to the value of“3” or “4”, the EP_stream_type indicates that the entry points are atconstant time intervals of 5 sec. With reference to such EP_stream_type,the playback device will recognize that stream analysis, if required,covering 5 seconds of data would be sufficient. In other words, theplayback device will be informed that any desired point can be accessedat random by analyzing at most five seconds data of the Out-of-MUXstream.

When set to either “6” or “7”, the EP_stream_type indicates that theentry points are located at the heads of DSs each composed of a completeset of functional segments. When starting playback, the playback devicereads functional segments starting from any of the entry points locatedat in constant intervals. Consequently, it is ensured that subtitles ora menu is presented starting from any intended playback point withoutstream analysis.

Since the EP_stream_type causes the playback device to judge whether upto five seconds of stream data needs to be analyzed or no streamanalysis is required at all. Thus, even if not only the Main Clip butalso the SubClip needs to be accessed at random, the playback device isnot subject to excessive burden. As a result of the reduced burden,response of the playback device to a user operation improves.

This concludes the description of the Clip information stored on thelocal storage 200.

<Structure of Local Storage 200: 3. PlayList Information>

Next, a description is given of PlayList information stored on the localstorage 200. Each file with the extension “mpls” (00001.mpls) storesPlayList (PL) information. PlayList information defines, as a PlayList(PL), two playback paths called “Main Path” and “SubPath” bundledtogether. FIG. 19 illustrates the data structure of PlayListinformation. As illustrated in the figure, the PlayList information iscomposed of: MainPath information (MainPath( )) defining a Main Path;PlayListMark information (PlayListMark( )) defining chapters; andSubPath information (SubPath( )) defining a SubPath.

A Main Path is a playback path defined on the main AV clip, whereas theSubPath is a playback path defined on the SubClip.

<PlayList Information: 1. MainPath Information>

First, the Main Path is described. The main path is a playback pathdefined on a video or audio stream carrying a main feature.

As indicated by arrows mp1, the Main Path includes a plurality of piecesof PlayItem information ( . . . PlayItem( )) . . . ). Each piece ofPlayItem information defines one or more logical units each specify aplayback section constituting the Main Path. As leader lines hs1indicate, the structure of a piece of PlayItem information isillustrated in detail. As illustrated, the PlayItem information iscomposed of the following fields: “Clip_information_file_name”indicating the name of a file containing the playback sectioninformation of the AV clip to which the In-point and Out-point of theplayback section belong; “Clip_codec_identifier” indicating the codingtype of the AV clip; and “In_time” which is timing informationindicating the start point of the playback section; and “Out_time” whichis timing information indicating the end point of the playback section.

FIG. 20 illustrates the relationship between the Main Clip and thePlayList information. The first level illustrates the timeline of thePlayList information. The second to fifth levels illustrate the videostream referenced to by the EP_map (identical to FIG. 5).

The PlayList information includes two pieces of PlayItem information,which are PlayItem Info #1 and PlayItem Info #2. The pairs of In_timeand Out_time in the respective pieces of PlayItem Info #1 and #2 definetwo playback sections. When aligning the two playback sections, atimeline that is different from the AV clip timeline is defined. Thistimeline is the PlayItem timeline illustrated on the first level. Asdescribed herein, by defining PlayItem information, a different timelinefrom the AV clip timeline is defined.

<Description of PlayList Information: 2. PlayListMark>

This concludes the description of PlayItem information according to theembodiment 1. Now, PlayListMark information will be described.

FIG. 21 illustrates the internal structure of PlayListMark informationincluded in the PlayList information. As indicated by leader lines pm0,the PlayListMark information is composed of a plurality of pieces ofPLMark information (PLMark #1 to PLMark #n). Each piece of PLMarkinformation (PLMark( )) specifies an arbitrary portion on the PLtimeline as a chapter point. As indicated by leader lines pm1, thePLMark information is composed of the following fields:“ref_to_PlayItem_id” indicating a PlayItem in which a chapter is to bedesignated; and “mark_time_stamp” specifying the position of a chapterin the PlayItem using time notation.

FIG. 22 illustrates chapter positions specified by the PLMarkinformation included in the PlayList information. The second to fifthlevels illustrate the EP_map and the AV clip identical to thoseillustrated in FIG. 20.

The first level in the figure illustrates the PlayListMark informationand the PL timeline. On the PL timeline, two pieces of PLMarkinformation #1 and #2 are present. Arrows kt1 and kt2 in the figurerepresent the designation by the ref_to_PlayItem_id. As shown by thearrows, the ref_to_PlayItem_id in the respective pieces of PLMarkinformation designate the respective pieces of PlayItem information. Inaddition, each Mark time stamp indicates a portion on the PlayItemtimeline to be designated as Chapter #1 and Chapter #2. As describedherein, PLMark information defines chapter points on the PlayItemtimeline.

<Description of PlayList Information: 3. SubPath Information>

As described above, the Main Path is a playback path defined within theMain Clip containing main feature video. On the other hand, the SubPathis a playback path defined within a SubClip to be played back insynchronism with the Main Path.

FIG. 23 illustrates the internal structure of SubPath information indetail. As arrows hc0 in the figure indicate, each SubPath informationincludes the following fields: “SubPath_type” indicating the type of theSubClip; and at least one piece of SubPlayItem information ( . . .SubPlayItem( ) . . . ).

In the figure, leader lines hc1 indicate that the structure ofSubPlayItem information is illustrated in detail. As indicated by arrowshc1, the SubPlayItem information is composed of the following fields:“Clip_information_file_name”, “SubPlayItem_In_Time”,“SubPlayItem_Out_Time”, “sync_PlayItem_id”, and“sync_start_PTS_of_PlayItem”.

The Clip_information file name specifies the name of file containingClip information, so that the SubClip associated with the SubPlayItem isuniquely identified.

The SubPlayItem_In_time indicates on the SubClip timeline the startpoint of the SubPlayItem.

The. SubPlayItem_Out_time indicates on the SubClip timeline the endpoint of the SubPlayItem.

The sync_PlayItem_id uniquely specifies one of the PlayItemsconstituting the Main Path, to be synchronized with the SubPlayItem. TheSubPlayItem_In_time is present on the playback timeline of the PlayItemspecified by the sync_PlayItem_id.

The sync_start_PTS_of_PlayItem specifies a point corresponding, on theplayback timeline of the PlayItem specified by the sync_PlayItem_id, tothe start point of the SubPlayItem specified by the SubPlayItem_In_time.

<SubPath Information: 1. SubPath_type>

This concludes the description of SubPath information. Next, theSubPath_type is described. The SubPath_type is set to a value from 0 to255 to indicate the type of the SubPath specified by the SubPathinformation.

The value of the SubPath_type is set in relation to the contents of theClip information specified by the Clip_information_file_name included inthe SubPlayItem information, and more specifically to theapplication_type included in the ClipInfo. For example, the SubPath_typeis set to a value from 0 to 255, and when the application_type of theClipInfo is set to “7”, the only possible values of the SubPath_type arefrom 5 to 8.

With the SubPath_type set to “5”, the SubPath information specifies aSubPath that is a playback path on the primary audio. The playback pathon the primary audio is defined for appending or replacing the primaryaudio playback defined by the PlayItem information.

With the SubPath_type set to “6”, the SubPath information specifies aplayback path on Presentation Graphics. The playback path on the PGstream according to the SubPath information is defined for appending orreplacing the PG stream defined by the PlayItem information.

With the SubPath_type set to “7”, the SubPath information specifies aplayback path on Interactive Graphics. The playback path on the IGstream is defined for appending or replacing the PG stream defined bythe PlayItem information.

With the SubPath_type set to “8”, the SubPath information specifies aplayback path on secondary audio. The playback path on the secondaryaudio stream is defined for audio mixing with the Primary audio definedby the PlayItem information.

This concludes the description of the SubPath_type.

<Details of SubPath Information: 2. Relationship Among Three>

The three used herein refers to the SubClip, the PlayList information,and the Main Clip. The SubClip and the PlayList information are bothstored on the local storage 200, whereas the Main Clip is stored on theBD-ROM.

FIG. 24 illustrates the relationship among the Main Clip stored on theBD-ROM and the SubClip and the PlayList information both stored on thelocal storage 200. In the figure, the first level illustrates theSubClips stored on the local storage 200. As illustrated, the SubClipsstored on the local storage 200 include the primary audio stream,secondary audio stream, PG stream, and IG stream. One of the SubStreamsspecified as the SubPath is supplied for synchronous playback with theMain Stream.

The second level illustrates the two timelines defined by the PlayListinformation. The lower one is the PlayItem timeline defined by thePlayItem information and the upper one is the SubPlayItem timelinedefined by the SubPlayItem.

As illustrated in the figure, the SubPlayItem_Clip_information_file_namespecifies one of the four SubClips illustrated on the first level and aportion of the specified SubClip is designated as a playback section.

The SubPlayItem_In_time and the SubPlayItem_Out_time define the startand end points of the playback section of the SubClip.

The sync_PlayItem_id represented by an arrow in the figure specifies aPlayItem to be synchronized with the SubClip. Thesync_start_PTS_of_PlayItem indicates the difference between the originpoints of the PlayItem timeline and the SubPlayItem timeline.

FIG. 25 illustrates the EP_map for the Main Clip, the PlayItem timeline,the EP_map for the SubClip, which is either a primary or secondary audiostream, and the SubPlayItem timeline.

In the figure, the middle level and the fourth to first levels from thebottom illustrate what is illustrated in FIG. 20. More specifically, thePlayItem timeline, the sequence of pictures, the MainClip timeline, theEP_map, and the sequence of TS packets are illustrated in the statedorder.

The first to third levels from the top illustrate what is illustrated inFIG. 16. More specifically, the TS packets, the EP_map, the SubCliptimeline are illustrated in the stated order. The fourth level from thetop illustrates the SubPlayItem timeline illustrated in FIG. 24. For theMain Clip, the entry times are set at time intervals of 1 second. Forthe SubClip, the entry times are set at time intervals of five seconds.

FIG. 26 illustrates the EP_map for the Main Clip, the PlayItem timeline,the EP_map for the SubClip, which is either a PG or IG stream.

In the figure, the fourth to first levels from the bottom illustratewhat is illustrated in FIG. 20. More specifically, the PlayItemtimeline, the sequence of pictures, the MainClip timeline, the EP_map,and the sequence of TS packets are illustrated in the stated order.

The first and third levels from the top illustrate what is illustratedin FIG. 16. More specifically, the sequence of TS packets, the EP_map,and the SubClip timeline are illustrated in the stated order. The fourthlevel from the top illustrates the SubPlayItem timeline illustrated inFIG. 24. For the Main Clip, the entry points are set at the timeintervals of one second. For the SubClip, the positions of DSs otherthan a Normal Case DS are set as entry points.

<Details of SubPath Information: 3. Interrelationship>

FIG. 27 shows a table of permissible combinations of the values of theSubPath_type, the application_type, and the EP_stream_type.

As shown in the table, when the SubPath_type is set to “5” and theapplication_type is set to “7”, the EP_stream_type is set to “3”. Sincethe EP_stream_type is set to “3”, the playback device will recognizethat the EP_map of this EP_stream_type is provided for primary audioplayback and thus the entry points are set at time intervals of fiveseconds or data intervals of 256K bytes.

In the case where the SubPath_type is set to “6” and theapplication_type is set to “7”, the EP_stream_type is set to “6”. Sincethe EP_stream_type is set to “6”, the playback device will recognizethat the EP_map of this EP_stream_type is provided for playback ofPresentation Graphics and thus the entry points are set at the heads ofDisplay Sets each composed of a complete set of functional segments.

In the case where the SubPath_type is set to “7” and theapplication_type is set to “7”, the EP_stream_type is set to “7”. Sincethe EP_stream_type is set to “7”, the playback device will recognizethat the EP_map of this EP_stream_type is provided for playback ofInteractive Graphics and thus the entry points are set at the heads ofDisplay Sets each composed of a complete set of functional segments.

In the case where the SubPath_type is set to “8” and theapplication_type is set to “7”, the EP_stream_type is set to “4”. Sincethe EP_stream_type is set to “4”, the playback device will recognizethat the EP_map of this EP_stream_type is provided for secondary audioplayback and thus the entry points are set at time intervals of fiveseconds or data intervals of 256K bytes.

This concludes the description of PlayList information, which is acomponent of the local storage 200. This concludes the overalldescription of the local storage 200.

<Virtual File System>

Now, a virtual file system will be described. FIG. 28 illustrates thevirtual file system created by a playback device 300. The top left blockof the figure illustrates the contents stored on the BD-ROM. The bottomleft block illustrates the contents stored on the local storage 200. Theright block illustrates the structure of the virtual file system.

The playback device creates the virtual file system through the combineduse of the AV clip, the clip information, the PlayList informationstored on the BD-ROM, with the AV clip, the Clip information, thePlayList information stored on the local storage 200.

More specifically, the virtual file system is created by:

i) adding the PlayList (00002.MPLS) on the local storage into the MPLSdirectory on the BD-ROM;

ii) adding the ClipInfo #2, #3, #4, and #5 (00002.CLPI, 00003.CLPI,00004.CLPI, and 00005.CLPI) on the local storage into the CLPI directoryon the BD-ROM; and

iii) adding the AV clip #2, #3, #4, and #5 (00002.M2TS, 00003.M2TS,00004.M2TS, and 00005.M2TS) on the local storage to the STREAM directoryon the BD-ROM.

As a result of the above processing, the virtual file system asillustrated in the right block of FIG. 28 is created.

This concludes the overall description of the recording medium accordingto the present invention. Next, a description is given of the playbackdevice according to the present invention.

<Internal Structure of Playback Device>

FIG. 29 illustrates the internal structure of the playback device. Basedon the internal structure illustrated in the figure, playback devicesconsistent with the present invention are industrially manufactured. Theplayback device of the present invention is roughly composed of twoparts, one of which is a system LSI and the other is a drive device. Bymounting those parts into a device cabinet and onto a substrate, theplayback device can be manufactured industrially. The system LSI is anintegrated circuit containing various processing units for implementingthe functions of the playback device. The playback device manufacturedin the above manner is composed of a BD drive 1, an arrival time clockcounter 2, a source de-packetizer 3, a PID filter 4, a transport buffer5, a multiplexed buffer 6, a coded picture buffer 7, a video decoder 8,a decoded picture buffer 10, a video plane 11, a transport buffer 12, acoded data buffer 13, a stream graphics processor 14, an object buffer15, a composition buffer 16, a composition controller 17, a presentationgraphics plane 18, a CLUT unit 19, a transport buffer 20, a coded databuffer 21, a stream graphics processor 22, an object buffer 23, acomposition buffer 24, a composition controller 25, an interactivegraphics plane 26, a CLUT unit 27, a compositor 28, a compositor 29, aswitch 30, a network device 31, a switch 32, an arrival time clockcounter 33, a source de-packetizer 34, a PID filter 35, a switch 36, atransport buffer 37, an elementary buffer 38, an audio decoder 39, atransport buffer 40, a buffer 41, an audio decoder 42, a mixer 43, ascenario memory 44, a controller 45, a main converter 46, asub-converter 47, a PL playback controller 48, and a PSR set 49. Notethat the internal structure illustrated in the figure is a decoder modelbased on the MPEG T-STD model and cabala of downconverting.

The BD drive 1 loads/ejects the BD-ROM and accesses the BD-ROM tosequentially reads Aligned Units each composed of 32 ES-TS packets.

The arrival time clock counter 2 generates an arrival time clock using a27 MHz crystal oscillator (27 MHz X-tal). The arrival timeclock is aclock signal defining the timeline on which the ATS assigned to each TSpacket is based.

Once an Aligned Unit composed of 32 EX-TS packets is read from theBD-ROM, the source de-packetizer 3 removes the TP_extra_header from eachof the 32 ES-TS packets and outputs the TS packets without the headersto the PID filter 4. The output by the source de-packetizer 3 to the PIDfilter 4 is performed at the timing when the time measured by thearrival time clock counter 2 reaches the ATS shown by theTP_extra_header. Since the output to the PID filter 4 is carried out inaccordance with the ATS, the TS packets are sequentially output to thePID filter 4 in accordance with the time measured by the arrival timeclock counter 2, regardless of the speed at which data is read from theBD-ROM, such as 1×-speed or 2×-speed.

The PID filter 4 judges, with reference to the PID attached to the TSpackets, the type of stream to which the TS packets belong is a videostream, a PG stream, an IG stream, or a primary audio stream. Accordingto the judgment, the PID filter 4 outputs the TS packets to one of thetransport buffer 5, the transport buffer 12, the transport buffer 20,and transport buffer 37.

The transport buffer (TB) 5 is a buffer for temporarily storing TSpackets output from the PID filter 4, if the TS packets belong to avideo stream.

The multiplexed buffer (MB) 6 is a buffer for temporarily storing PESpackets output from the transport buffer 5, in order to later output thevideo stream to the coded picture buffer 7.

The coded picture buffer (CPB) 7 is a buffer for storing coded pictures(I pictures, B pictures, and P pictures).

The video decoder 8 decodes individual frames contained in the videoelementary stream at every predetermined decoding time (DTS) to obtain aplurality of frames and renders the resulting picture data on thedecoded picture buffer 10.

The decoded picture buffer 10 is a buffer on which decoded picture datais rendered.

The video plane 11 is used for presenting uncompressed picture data. Aplane is a memory area of the playback device for storing a frame ofpixel value data. The video plane 11 stores picture data at theresolution of 1920×1080, and the picture data is composed of pixelvalues each expressed by 16-bit YUV values.

The transport buffer (TB) 12 is a buffer for temporarily storing TSoutput from the PID filter 4, if the TS packets belong to a PG stream.

The coded data buffer (CDB) 13 temporarily stores PES packetsconstituting a PG stream.

The stream graphics processor (SPG) 14 decodes an ODS to obtainuncompressed graphics data expressed by index colors, and renders theobtained graphics data as a graphics object on the object buffer 15.Decoding by the stream graphics processor 14 is instantaneous, and thestream graphics processor 14 temporarily stores the resulting graphicsobject. Although the decoding by the stream graphics processor 14 isinstantaneous, transfer of the graphics object to the object buffer 15takes some time. In the player model conforming to the BD-ROMspecification, the transfer rate to the object buffer 15 is 128 Mbps.The end time of transfer to the object buffer 15 is shown by the PTS inthe END segment. Thus, until the time shown by the PTS is reached, theprocessing on the subsequent DS is delayed. The rendering of thegraphics object obtained by decoding each ODS is started at the timeshown by the DTS associated with that ODS and end by the time shown bythe PTS associated with that ODS.

The object buffer 15 holds the graphics object obtained by decoding anODS by the stream graphics processor 14. The object buffer 15 needs tobe twice or four times as large as the graphics plane 18 for thefollowing reason. For implementing scrolling, it is necessary to store agraphics object that is two or four times as large as the graphics plane18.

The composition buffer 16 is memory for storing PCS and PDS. In the casewhere there are two DSs to be processed and where the active period oftheir PCS overlap, the composition buffer 16 stores the plurality of PCSto be processed.

The composition controller 17 judges, each time the current playbackpoint reaches a new DS, whether the Composition_state of the PCScontained in the new DS indicates Epoch Start, Acquisition Point, orNormal Case. If the Epoch Start is indicated, the composition controller17 transfers the PCS on the coded data buffer 13 to the compositionbuffer 16.

The presentation graphic plane 18 is a memory area as large as one fullscreen and stores uncompressed graphics data worth one screen. Thepresentation graphic plane 18 stores uncompressed graphics data at theresolution of 1920×1080 and the uncompressed graphics data is composedof pixel values each expressed using an 8-bit index colors. Byconverting the index colors with reference to a CLUT (Color LookupTable), the uncompressed graphics data stored on the presentationgraphics plane 18 is supplied for display.

The CLUT unit 19 converts the index colors of the uncompressed graphicsdata stored on the presentation graphic plane 18 to Y, CR, and Cbvalues.

The transport buffer (TB) 20 is a buffer for temporarily storing TSpackets belonging to an IG stream.

The coded data buffer (CDB) 21 is a buffer for temporarily storing PESpackets constituting an IG stream.

The stream graphics processor (SPG) 22 decodes an ODS and transfersuncompressed graphics data obtained by the decoding to the object buffer23.

The object buffer 23 stores a plurality of uncompressed graphics objectsdecoded by the stream graphics processor 22. The rectangle area of theobject buffer 23 to be occupied by each graphics object is identifiedwith reference to the object_id included in the associated ODS.Consequently, when a graphics object enters the object buffer 23 and theobject_id of the graphics object is the same as that of the graphicsobject currently on the object buffer 23, the area occupied by thecurrent graphics object is overwritten with the new graphics object.

The composition buffer 24 is a buffer for storingInteractive_composition transported by one or more ICSs. TheInteractive_composition stored on the composition buffer 24 is suppliedto the composition controller 25 for decoding.

The composition controller 25 judges, each time the current playbackpoint reaches a new DS, whether the composition_state of the ICScontained in the new DS indicates Epoch Start, Acquisition Point, orNormal Case. If the Epoch Start is indicated, the composition controller25 transfers the new interactive_composition on the coded data buffer 2lto the composition buffer 24.

Each time the ICS contained in an acquisition point DS is supplied tothe coded data buffer 21, the composition controller 25 compares thepage information of the newly supplied ICS with the page information ofthe interactive_composition already stored in the composition buffer 24.If the page information with a greater value page_version_number ispresent on the coded data buffer 21, the page information is transferredto the composition buffer 24. As a result, the intended page informationon the composition buffer 24 is updated. The composition controller 25further judges whether the thus updated page information is currentlybeing displayed. If the updated page is being displayed, the page isrendered again to reflect the update.

The interactive graphics plane 26 is used for presenting uncompressedgraphics data obtained by decoding graphics object by the streamgraphics processor (SGP) 22. The graphics data is rendered on theinteractive graphics plane 26 at the resolution of 1920×1080, and thegraphics data is composed of pixel values each expressed using 8-bitindex colors. By converting the index colors with reference to the CLUT(Color Lookup Table), the uncompressed graphics data stored on theinteractive graphics plane 26 is supplied for presentation.

The CLUT unit 27 converts the index colors of the uncompressed graphicsdata stored on the interactive graphics plane 26 to Y, CR, and Cbvalues.

The compositor 28 overlays the uncompressed frame data rendered on thevideo plane 11 with the uncompressed graphics object rendered on thepresentation graphic plane 18. As a result of the overlaying, thecomposite image in which text subtitles are overlaid on video isobtained.

The compositor 29 overlays the uncompressed graphics object rendered onthe interactive graphics plane 26 with the composite image (uncompressedpicture data overlaid with the uncompressed graphics object rendered onthe presentation graphic plane 18) output from the compositor 28.

The switch 30 selectively supplies the TS packets read from the BD-ROMor the TS packets read from the local storage to the transport buffer20.

The network device 31 is used to implement communications functionalityof the playback device. More specifically, the network device 31establishes TCP connection, FTF connection, and so on with a web site atan URL. The contents downloaded from the web site via the connectionestablished by the network device 31 are stored to the local storage200.

The switch 32 selectively supplies the TS packets read from the BD-ROMor the TS packets read from the local storage to the transport buffer12.

The source de-packetizer 34 removes the TP_extra_header from each TSpacket constituting the AV clip read from the local storage 200 andoutputs the TS packets without headers to the PID filter 35. The outputof the TS packets to the PID filter 35 is carried out at the timing whenthe time measured by the arrival time clock counter 33 reaches the ATSshown by the TP_extra_header.

The PID filter 35 switches to output the TS packets read from the localstorage 200 toward either of the PG stream decoder, IG stream decoder,the audio decoder 39, and the audio decoder 42.

The switch 36 supplies toward the audio decoder 39 the TS packets readfrom the BD-ROM or from the local storage 200. The TS packets constitutethe primary audio stream. By the presence of the switch 36, the primaryaudio stream can be supplied to the audio decoder 39 from either of theBD-ROM and the local storage 200.

The transport buffer (TB) 37 stores TS packets carrying the primaryaudio stream.

The elementary buffer (EB) 38 stores the PES packets carrying theprimary audio stream.

The audio decoder 39 decodes the PES packets carrying the primary audiostream output from the elementary buffer 41 and outputs uncompressedaudio data.

The transport buffer (TB) 40 stores TS packets carrying the secondaryaudio stream.

The elementary buffer (EB) 41 stores PES packets carrying the secondaryaudio stream.

The audio decoder 42 decodes the PES packets carrying the secondaryaudio stream output from the elementary buffer 38 and outputsuncompressed audio data.

The mixer 43 mixes the uncompressed audio data obtained by decoding theprimary audio stream, with the uncompressed audio data obtained bydecoding the secondary audio stream. The mixer 43 then outputs thecomposite audio resulting from the mixing.

The scenario memory 44 stores current PlayList information and currentClip information. The current PlayList information used herein refers tothe currently processed PlayList information from among a plurality ofpieces of PlayList information stored on the BD-ROM. The current Clipinformation used herein refers to the currently processed Clipinformation from among a plurality of pieces of Clip information storedon the BD-ROM.

The controller 45 is composed of an instruction ROM and CPU and executessoftware stored on the instruction ROM to carry out overall control ofthe playback device. The control executed on the playback devicedynamically changes in response to a user event generated upon receiptof a user operation and in accordance with the values held in each PSRof the PSR set 49. The controller 45 is composed of functional elementsof the main converter 46, the sub-converter 47, and the PL playbackcontroller 48.

The main converter 46 converts playback points on the PlayItem timelineto corresponding addresses on the Main Clip. The playback points forstarting normal playback and trick play are defined on the PlayItemtimeline. With reference to the EP_map included in the Clip informationassociated with the MainClip, the main converter 46 converts theplayback points to the corresponding addresses on the Main Clip.

The sub-converter 47 first converts the playback points on the PlayItemtimeline to corresponding playback points on the SubPlayItem timeline,and further converts the resulting playback points to correspondingaddresses on the SubClip. The conversion of the playback points by thesub-converter 47 is carried out with reference to the sync_PlayItem_idand the sync_start_PTS_of_PlayItem included in the SubPlayIteminformation.

The playback points on the SubPlayItem timeline are converted tocorresponding addresses on the SubClip through the following twoprocesses:

(i) determining the entry point that is near the random access pointform among a plurality of entry points shown by the EP_map; and

(ii) performing the stream analysis starting from the thus determinedentry point.

Note that the process (i) is carried out using the EP_map for theSubClip. The process (ii) (i.e., stream analysis) is necessary forallowing random access in the case where the SubClip is the primary orsecondary audio stream. In the case where the SubClip is the PG or IGstream, however, the stream analysis may be omitted and random access tothe SubClip is still ensured. Whether it is possible to omit the processof stream analysis is determined with reference to the EP_stream_type.The stream analysis may be omitted because the Clip information containsEP_map that ensures the stream to be suitably accessed at random.

The PL playback controller 48 carries out overall control of theplayback device for PL playback. The PL playback used herein refers tosynchronous playback of the Main Clip stored on the BD-ROM and theSubClip stored on the local storage, in accordance with the PlayIteminformation and SubPlayItem information contained in the PlayListinformation. The Main Clip and SubClip are associated with their ownEP_map, so that high-speed random access to the respective AV clips isensured. During PL playback, the “normal playback” and the “trick play”are implemented through high-speed random access to the respectiveclips.

The normal playback during PL playback used herein refers to playback ofspecific portions of the Main Clip and SubClip on the PlayItem timelineand the SubPlayItem timeline, respectively. The portions of therespective clips are specified by the PlayItem information defining theMainPath information and SubPlayItem information defining the SubPathinformation.

The trick play during PL used herein refers to fast-forward playback,rewind playback, chapter-search playback, and time-search playback ofspecific portions of the MainClip and SubClip. The portions of therespective clips are specified by the PlayItem information defining theMainPath information and the PlayItem information defining the SubPathinformation.

The PSR set 49 is a set of non-volatile registers provided within theplayback device. The set of registers include 64 player status registers(PSR(1)-PSR(64)) and 4,096 general-purpose registers (GPRs). The 64player status registers (PSRs) represent the current status of theplayback device, including the current playback point. The currentplayback point is represented by the values of PSR(5)-PSR(8).Specifically, PSR(5) is set to a value from 1-999 to indicate thechapter number to which the current playback point belongs. When set to“0xFFFF”, PSR(5) indicates that the chapter numbers are invalid in theplayback device.

PSR(6) is set to a value from 0-999 and indicates the PL number to whichthe current playback point belongs (current PL Number).

PSR(7) is set to a value from 0-255 and indicates the PlayItem number towhich the current playback point belongs (current PI Number).

PSR(8) is set to a value from 0-0xFFFFFFFF and indicates the currentplayback point (current PTM) in 45 kHz accuracy.

This concludes the description of the internal structure of the playbackdevice.

<Software Implementation: 1. PL Playback Controller 48>

Hereinafter, a description is given of software implementation of the PLplayback controller 48. The PL playback controller 48 can be implementedas computer software by creating a program for causing a computer toexecute the processing steps illustrated in FIG. 30. Both the normalplayback and trick play during PL Playback are executed through a jumpplayback in which playback is started at an arbitrary point on thePlayItem timeline. Thus, the PL playback is executed by instructing theplayback device to execute jump playback.

FIG. 30 is a flowchart illustrating the processing steps executed by theplayback device for jump playback in which playback is started from anarbitrary point on the PlayItem timeline. First, the playback devicecalculates an offset a that indicates a coordinate of a point on thePlayItem timeline (Step S1). Next, the offset α is converted to acoordinate of a point on the MainClip timeline (In_time+offset α) (StepS2). Then, the coordinate on the MainClip timeline (In_time+offset α) isconverted to a corresponding address a using the EP_map associated withthe Main Clip (Step S3).

In a step S4, the playback device judges whether the EP_stream_type inthe EP_map associated with the SubClip is set to any of the values 3, 4,6, and 7. If the EP_stream_type is not set to any of the above values,the playback device reads the BD-ROM starting from the address α (stepS9). On the other hand, if the EP_stream_type is set to any of 3, 4, 6,and 7, the playback device converts the offset α to an offset β (=offsetα−sync_start_PTS_of_PlayItem) indicating a coordinate of a correspondingpoint on the SubPlayItem timeline (step S5), and then converts theoffset β to a coordinate of a corresponding point on the SubCliptimeline (SubPlayItem_In_time+offset β) (Step S6). Then, with referenceto the EP_map, the playback device converts the thus obtained coordinate(SubPlayItem_In_time+offset β) to a corresponding address β (step S7).Then, the playback device simultaneously reads the BD-ROM starting fromthe address α and the local storage 200 starting from the address β(step S8).

If the EP_stream_type is set to a value other than 3, 4, 6, and 7 (stepS4: No), it means that the EP_stream_type is invalid. When theEP_stream_type is invalid, the playback device is provided with noinformation as to the intervals between entry points or the intervalsbetween entry times in the Out-of-MAX stream. With this being thesituation, it is possible that it takes long to determine an accesspoint on the SubClip and thus takes long to make random access to theSubClip. In such a case, synchronous playback of the Main Clip andSubClip may result in significant delay in the Main Clip playback. Inorder to avoid such undesirable possibility, the playback deviceprohibits random access to the SubClip, and only reads the Main Clipstarting from the address α (step S9). Since random access is made onlyto the Main Clip, there will be no significant processing delay due torandom access.

FIG. 31 schematically illustrate how a random access point is determinedusing the EP_map illustrated in FIG. 25. The following describes, withreference to FIG. 31, the processing steps performed to specify therandom access point illustrated in FIG. 31. Note that FIG. 31illustrates the case where the requested jump point happens to coincidewith an entry point of both the Main Clip and SubClip. Suppose that therequested jump point is at the offset a on the PlayItem timeline, theorigin point of the PlayList timeline is located at the point on theMainClip timeline specified by In_time. Thus, the jump point on the MainClip is calculated by “In_time+offset α”. Thus the calculated result isconverted to a corresponding SPN, and the SPN is further converted to acorresponding sector number.

On the other hand, the point on the SubPlayItem timeline correspondingto the offset a on the PlayItem timeline is specified bysync_start_PTS_of_PlayItem+offset β. It is because the origin point ofthe SubPlayItem timeline is apart from the origin point of the PlayItemtimeline by the distance indicated by sync_start_PTS_of_PlayItem. Oncethe offset β is calculated in the above manner, the coordinate of thejump point on the SubClip timeline is calculated. Since the origin pointof the SubPlayItem timeline is located at the point on the SubPlayItemtimeline specified by In_time, the jump point is calculated by“SubPlayItem_In_time+offset β”. Then, the jump point is converted into acorresponding SPN using the EP_map of the SubClip, and the SPN isconverted to a corresponding address.

<Software Implementation: 2. Main Converter 46 & Sub-Converter 47>

Hereinafter, a description is given of software implementation of themain converter 46 and sub-converter 47. The main converter 46 andsub-converter 47 can be implemented as computer software by creating aprogram for causing a computer to execute the processing stepsillustrated in FIG. 32.

FIG. 32 is a flowchart illustrating the processing steps executed by theplayback device for converting a coordinate TM on the Main Clip and theSubClip to corresponding addresses.

The playback device keeps adding the time length specified by eachPTS_EP_High included in the EP_High until the total Σ of the timelengths first exceeds the In_time, to see how many PTS_High fields areadded (Step S11). The time length indicated by each PTS_EP_High is atime unit whose most significant bits are indicated by the PTS_EP_High.The playback device then identifies the EP_High_id of the k-th EP_Highthat results in the total Σ exceeding the In_time and stores the valueof k (step S12).

Next, the playback device keeps adding, to the total Σ of up to thePTS_EP_High(k'11), the time length indicated by each PTS_EP_Low includedin EP_Low until the total first exceeds In_time, to see how many PTS_Lowfields can be added (step S13). The playback device then determines theEP_Low_id of the h-th EP_Low that results in the total exceeding the Intime and stores the value of h (step S12).

Using the pair of values “k−1” and “h−1” obtained as above, the playbackdevice specifies the entry point that is near the In_time. Using theSPN_EP_High and SPN_EP_Low of the thus specified entry point, theplayback device calculates the SPN that coincides with the coordinate TMand then converts the SPN to a corresponding sector address (step S15).

Conversion of SPN into a corresponding sector address is performedthrough the following processes. SPNs are serial numbers uniquelyassigned to individual TS packets. Thus, for a playback device to readthe TS packet corresponding to a specific SPN, it is necessary toconvert the SPN to a relative sector number. Here, as illustrated inFIG. 4, every 32 TS packets are converted into one Aligned Unit andrecorded into three sectors. Thus, the result obtained by dividing theSPN by 32 is the number assigned to the Aligned Unit containing anI-picture. Then, the result obtained by multiplying the thus obtainedAligned Unit number by 3 is the sector address of the Aligned Unit thatis near the SPN. The sector address calculated in the above manner is arelative sector number serially assigned to each sector from the head ofone AV clip. Thus, by setting the file pointer to the relative sectornumber, the sector corresponding to the entry point is specified.

In a step S16, it is determined whether the EP_stream_type of the EP_mapfor the SubClip is set to any of the values 3, 4, 6, and 7. In the casewhere the EP_stream_type is set to 6 (PG stream) or 7 (IG stream), itmeans that the entry point near the coordinate TM coincides with thehead (PCS or ICS) of a DS other than Normal Case DS. Since such a DScontains all the functional segments necessary for presenting acomposite image, the address obtained by the conversion in the step S15is designated as the address β, which is the access point (step S17).

Once the address β is determined, the playback device starts playbackfrom the address β to duly present text subtitles or menus.

In the case where the EP_stream_type is set to 3 (primary audio stream)or 4 (secondary audio stream), the playback device performs streamanalysis starting from the address obtained by the conversion in thestep S15 (step S18). In the stream analysis, the PES packet header isextracted and the audio frame size and PTS are read from the header. Thepoint at which the next audio frame is recorded is identified based onthe size. The above process is repeated until the thus determined pointreaches the PES packet with the PTS set to a value greater than thecoordinate TM (step S19).

Once the PES packet with the PTS set to a value greater than thecoordinate TM is identified, the address of the PES packet is determinedas the address β, which is the access point (step S20). The playbackdevice then starts playback from the address β to duly present audio.

Hereinafter, the processing steps illustrated in the flowchart are morespecifically described with reference to FIGS. 33 and 34.

FIG. 33 illustrates the relationship among the variables k and h and therandom access point, in the case where the SubClip is a primary orsecondary audio stream. In the figure, the first level illustrates thePTS_EP_High fields included in the EP_map. The second revel illustratesthe PTS_EP_Low fields also included in the EP_map. The third levelillustrates TS packets. In the figure, the k-th PTS_EP_High has theminimum value exceeding the random access point, and the h-th PTS_EP_Lowhas the minimum value exceeding the random access point.

Thus, the pair of PTS_EP_High[k−1] and the PTS_EP_Low[h−1] specifies theentry time near the random access point.

Since the above pair of the PTS_EP_High and the PTS_EP_Low specifies theentry point near the random access point, the PES packet correspondingto the random access point is specified through the stream analysisstarted from the thus specified entry point.

FIG. 34 illustrates the relationship among the variables k and h and therandom access point, in the case where the SubClip is a PG or IG stream.In the figure, the first level illustrates the PTS_EP_High fieldsincluded in the EP_map. The second revel illustrates the PTS_EP_Lowfields also included in the EP_map. The third level illustrates TSpackets. In the figure, the k-th PTS_EP_High has the minimum valueexceeding the random access point, and the h-th PTS_EP_Low has theminimum value exceeding the random access point.

Thus, the pair of PTS_EP_High[k−1] and the PTS_EP_Low[h−1] specifies theentry time near the random access point.

As stated above, the above pair of the PTS_EP_High and the PTS_EP_Lowspecifies t-he entry point near the random access point. Thus, as longas the playback device starts reading from the thus specified entrypoint, it is always ensured that a DS composed of a complete set offunctional segments is read.

As described above, according to the present embodiment, the EP_map forthe SubClip includes the EP_stream_type. The provision of EP_stream_typeallows the playback device to be informed of whether the EP_map for theOut-of-MUX stream shows entry points set at constant time intervals oraddresses of independent playback sections. Being informed of the typeof the EP_map, the playback device is enabled to instantly judge whetherhigh-speed random access to the SubClip is possible. With thisarrangement, synchronous playback of the Main Clip and SubClip isexecuted without sacrificing a response performance. Thus, even duringPlayList playback defining the Main Path+SubPath, jump playback isexecuted at the same level of response speed as that of jump playbackexecuted solely on the Main Path.

Second Embodiment

A second embodiment of the present invention relates to improvements inPicture in Picture (PiP). The PiP playback refers to playback of suchPlayList information that includes MainPath information and SubPlayIteminformation. In addition, the MainPath information specifies a Main Clipthat contains video, and the SubPath information specifies a SubClipthat contains different video. When PiP playback of the PlayListinformation is executed, video playback of the SubClip (secondary video)is presented within video playback of the Main Clip (primary video) onthe same screen.

FIG. 35 illustrates one example of PiP playback. In the figure, untilthe current playback point reaches a predetermined point tp, playback ofthe primary video is solely presented. After the point tp is reached,playback images of the secondary video are presented within playbackimages of the primary video. In this example, the primary video ispresented with HD (high-definition) images, whereas the secondary videois presented with SD (standard-definition) images.

In this example, the secondary video presents video images in which thedirector or a performer of a movie points with his finger a specificobject appearing in the primary video. When the secondary video ispresented in combination with the primary video playback, the combinedimages present a scene in which the director or the performer points outwith his finger the specific object appearing in playback images of theprimary video and gives commentary about the object.

FIG. 36A illustrates an HD image and an SD image for comparison.

The HD image has the resolution of 1920×1080. The frame interval is a3750 clock (alternatively, a 3753 or 3754 clock), which is equivalent tothe frame interval of a theatrical movie film.

The SD image is has the resolution of 720×480. The frame interval is a1501 clock, which is equivalent to the frame interval of NTSC.Alternatively, the frame interval may be an 1800 clock, which isequivalent to the frame interval of PAL.

As illustrated in the figure, the SD image resolution is about ¼ of theHD image resolution. Thus, when the primary video (i.e. HD images) andthe secondary video (i.e. the SD images) are presented on the samescreen, the size of the secondary video on screen is about ¼ of theprimary video. FIG. 36B illustrates how the secondary video is scaled upand down. The secondary video is scaled up and down in accordance withthe scaling factor. The scaling factor defines ×¼ height, ×½ height,×1.5 height, or ×2.0 height. The playback device reduces or enlarges thesecondary video in height according to the scaling factor. At the sametime, the playback device also reduces or enlarges the secondary videowidthwise, so as to maintain the aspect ratio of the original SD image.

Through the scaling up and scaling down of the secondary video, thelayout of the secondary video in PiP playback is freely changed.

Now, the following describes the structures of the recording medium andthe playback device for creating a PiP playback application describedabove. In this embodiment, the AV clip stored on the BD-ROM constitutesprimary video. It is simply because a large volume medium is suitablefor distribution of the primary video having a large volume of data. Onthe other hand, the SubClip constituting secondary video is distributedto the playback device via a network and stored in the local storage,along with the PlayList information defining synchronous playback of theprimary video and the secondary video.

<Structure of Local Storage 200: 1. File Structure>

FIG. 37 illustrates the contents stored on the local storage accordingto the second embodiment. The structure of the local storage isillustrated in FIG. 37 in the similar manner to FIG. 11. The differencewith FIG. 11 is found in that the SubClip (00002.m2ts) is a secondaryvideo stream and the clip information (00002.clip) contains the EP_mapfor the secondary video stream.

<Structure of Local Storage 200: 2. Clip Information>

FIG. 38 illustrates the internal structure of the clip informationstored on the local storage. The clip information is associated with thesecondary video stream. In the figure, leader lines cu2, cu3, cu4, andcu5 indicate that the internal structure of the EP map included in theclip information is illustrated in more detail.

The EP_map pointed to by the leader lines is identical in structure tothe EP_map illustrated in FIG. 8. The EP_map of the secondary videoshows entry points set on the head of each access unit (GOP)constituting the secondary video stream, and also shows entry timescorresponding to the entry points. Although for the secondary video, theEP_map is still set for video. Thus, the time interval between adjacententry times is less than one second, and the EP_stream_type is set tothe value “1: Video Type” similarly to the example shown in FIG. 8.

The leader lines ct1 and ct2 indicate that the internal structure ofClip Info is illustrated in more detail. In the Clip Info, theapplication type is set to the value indicating that the SubClipassociated with the clip information is “TS for additional content withvideo”. FIG. 39 illustrates the EP_map set for the secondary videostream in the same manner as FIG. 9.

This concludes the description of the clip information according to thesecond embodiment.

<Structure of Local Storage 200: 3. PlayList Information>

Next, the following describes the PlayList information according to thesecond embodiment. There are two types of PiP playback; one isstatically synchronized playback and the other is dynamicallysynchronized playback. The above-mentioned example is a staticallysynchronized PiP application for synchronously presenting the mainfeature movie with the commentary, which are the primary video and thesecondary video, respectively.

<Details of PlayList Information: 1. Components for Implementing StaticPiP Playback>

FIG. 40 illustrates PlayList information defining staticallysynchronized PiP playback. The PlayList information defining thestatically synchronized PiP playback may include a plurality of piecesof SubPath information (Multi-SubPath) and each piece of SubPathinformation may include a plurality of pieces of SubPlayItem information(Multi-SubPlayItem).

The SubPlayItem information illustrated in the figure additionallyincludes a PiP_position field and a PiP_size field. Each field includedin the SubPlayItem information is set as follows.

The “Clip_information_file_name” field is set to indicate the file nameof the AV clip containing the primary video.

The “sub_PlayItem_In_time” field is set to the value indicating thestart point of the SubPlayItem on the timeline of the SubClip containingthe secondary video.

The “sub_PlayItem_Out_time” field is set to the value indicating the endtime of the SubPlayItem on the timeline of the SubClip containing thesecondary video.

The “sync_PlayItem_id” field is associated with the AV clip containingthe primary video and set to the value indicating the PlayItemidentifier.

The “sync_start_PTS_of_PlayItem” field is timing information indicatingthe timing at which playback of the playback section (SubPlayItem)defined by the SubPlayItem information is to be started. Here, theplayback start time of SubPlayItem is expressed using the number ofseconds to be lapsed from the playback start of the PlayItem defined bythe sync_PlayItem_id.

The “PiP_position” indicates the X and Y coordinates of a position inthe screen plane at which the secondary video playback is to bepresented.

The “PiP_size” indicates the height and the width of the secondary videoplayback.

<Details of PlayList Information: 2. Synchronization in StaticallySynchronized PiP Playback>

FIG. 41 illustrates how the PlayList information defines thesynchronization between the Main Clip containing the primary video andthe SubClip containing the secondary video. FIG. 41 is illustrated inthe same manner as FIGS. 25 and 26.

The SubClip containing the secondary video is provided with an EP_mapand thus the high-speed random access to the SubClip is ensured in timeaccuracy of less than one second. The PlayList information definessynchronous playback of the Main Clip and SubClip on precondition thathigh-speed random access to the SubClip is possible.

Here, the following describes the technical significance of theprovision of EP_map for a SubClip containing secondary video. TheSubClip containing secondary video is provided with an EP map becausetrick play of the secondary video needs to be executed in response totrick play of the primary video. In the case where trick play of theprimary video is executed during PiP playback, how to process thesecondary video matters. During fast-forwarding or rewinding of theprimary video, it is possible to continually play back the secondaryvideo in the normal mode. Yet, in such a case, the playback device needsto have separate decoders for the primary video and the secondary video,and the decoders need to be operated on separate system time clocks(STCs). It is impractical to provide two STCs to a player modelprimarily intended for home use.

Practically, the secondary video is fast-forwarded and rewound insynchronism with fast-forwarding and rewinding of the primary video. Inview of this, the EP_map is also provided for the SubClip so as to allowthe secondary video to be fast-forwarded and rewound in synchronism withfast-forwarding and rewinding of the primary video.

This concludes the description of the PlayList information definingstatically synchronized playback of the Main Clip and SubClip.

<Details of PlayList Information: 3. Synchronization in DynamicallySynchronized PiP Playback>

Next, a description is given of the PlayList information definingdynamically synchronized playback of the Main Clip and SubClip.

The dynamic synchronization refers to playback of the SubPlayItemassociated with the secondary video stream (SubClip) synchronously withplayback of the primary video stream (Main Clip), and the playback startpoint of the SubPlayItem is dynamically determined in response to a useroperation. For example, the author of a PiP application intends toimplement such dynamic synchronization as illustrated in FIGS. 42Athrough 42C.

FIG. 42A illustrates the display screen presented during playback of theprimary video of the PiP application. Each button on the screenillustrated in FIG. 42A is presented using a thumbnail image (thumbnailbutton). Each thumbnail button has three states that are a normal state,a selected state, and an activated stated. The PiP application isdesigned so as to present, when one of the thumbnail buttons receivesthe selected state (FIG. 42B), playback image of the secondary video inplace of the thumbnail image used to present the selected button (FIG.42C). As in this example, the playback start point of the secondaryvideo is dynamically determined in response to a user operation madeduring playback of the primary video. This is what is referred to as“dynamic synchronization”. A user operation for changing a thumbnailbutton to the selected state (in this case, a user operation forselecting the thumbnail button) is referred to as a rock operation.

In the case of dynamic synchronization, it is not known in advance whichof the thumbnail images will be selected and thus which secondary videois to be played back. Thus, it is necessary to dynamically determine,during playback of the primary video, the start point of the synchronousplayback of the secondary video.

<Details of PlayList Information: 4. Components for ImplementingDynamically Synchronized PiP Playback>

FIG. 43 illustrates the internal structure of the PlayList informationdefining dynamically synchronized PiP playback.

The PlayList information defining the dynamically synchronized PiPplayback may include a plurality of pieces of SubPath information(Multi-SubPath). Yet, each piece of SubPath information can include asingle piece of SubPlayItem information (Single-SubPlayItem).

Similarly to the example illustrated in FIG. 40, the SubPlayIteminformation illustrated in the figure includes a PiP_position field anda PiP_size field. The setting of “Clip_information_file_name”,“SubPlayItem_In_time”, “SubPlayItem_Out_time”, “sync_PlayItem_id”,“PiP_position”, and “PiP_size” fields is identical to the setting ofcorresponding fields illustrated in FIG. 40. The difference lies in thesetting of the “sync_start_PTS_of_PlayItem”.

The “sync_start_PTS_of_PlayItem” indicating a point for establishingsynchronous playback with the PlayItem specified by the sync_PlayItem_idis set to an undefined value. If set to an undefined value, the“sync_start_PTS_of_PlayItem” indicates that the synchronization point isdynamically determined, in response to a rock operation by a user, onthe timeline of the PlayItem specified by the sync_PlayItem_id.

In accordance with the PlayList information illustrated in FIG. 43, theplayback device operates to write the point of a rock operation to thesync_start_PTS_of_PlayItem. Here, if the point of rock operation isdefined as a time at which a thumbnail button is first changed to theselected state and then automatically to the activated state, theplayback device starts playback of the secondary video at the time whenthe thumbnail button is changed to the activated state. Since this pressrelates specifically to the selection and automatic activation ofthumbnail buttons, it is desirable to define this process using a navicommand, which is used to control the behavior of thumbnail buttons.

This concludes the description of the internal structure of the PlayListinformation implementing dynamically synchronized PiP playback. Thisconcludes the description of the improvements made on the recordingmedium, according to the present embodiment.

Now, the following describes improvements made on the playback device,according to this embodiment.

<Structure of Playback Device: 1. Overall Structure>

FIG. 44 illustrates the internal structure of the playback deviceaccording to the second embodiment. For the sake of convenience inillustration, the components of the audio decoder are omitted.

FIG. 44 is based on the structure of the playback device illustrated inFIG. 29, and the same reference numerals are used to denote the samecomponents. Among the components in common with the playback deviceillustrated in FIG. 29, the transport buffer 5, the multiplexed buffer6, the coded picture buffer 7, the video decoder 8, the decoded picturebuffer 10, and the primary video plane 11 in the second embodimentfunction to decode the primary video stream.

On the other hand, for decoding the secondary video stream, the playbackdevice illustrated in FIG. 44 additionally includes a transport buffer51, a multiplexed buffer 52, a coded picture buffer 53, a video decoder54, a decoded picture buffer 55, a secondary video plane 56, a scaler57, and a compositor 58. Hereinafter, a description of those additionalcomponents will be made.

<Structure of Playback Device 300: 2. Additional Hardware Components>

The transport buffer 51 is a buffer for temporarily storing TS packetscarrying the secondary video stream (SubClip) output from the PID filter35.

The multiplexed buffer (MB) 52 is a buffer for temporarily storing PESpackets output from the transport buffer 51, in order to later outputthe secondary video stream to the coded picture buffer 53.

The coded picture buffer (CPB) 53 is a buffer for storing encodedpictures (I, B, and P pictures).

The video decoder 54 decodes individual frames contained in thesecondary video stream at every decoding time shown by a decoding timestamp (DTS) to obtain a plurality of frames and renders the resultingpicture data on the decoded picture buffer 55.

The decoded picture buffer 55 is a buffer on which decoded picture datais rendered.

The secondary video plane 56 is used for presenting unimpressed picturedata obtained by decoding the secondary video.

The scaler 57 is for reducing or enlarging the size of playback imagerendered on the secondary video plane 56, in accordance with the heightand the width indicated by the PiP_size field included in theSubPlayItem information.

The compositor 58 is for implementing PiP playback by overlaying theplayback image resized by the scaler 57, on the playback image obtainedby the video decoder. The overlaying of the playback images of theprimary video and the secondary video is carried out in accordance withthe PiP_position defined in the SubPlayItem information. As a result,composite images of the primary and secondary video playback arepresented. The compositor 58 is capable of chroma key composition, layercomposition, and so on. Thus, it is possible to remove the backgroundimage from the secondary video to only leave a film character and thenoverlay the resulting image of the film character with the playbackimage of the primary video.

With the above additional components, the PID filter 35 according to thesecond embodiment supplies TS packets carrying the secondary videostream to the transport buffer 51, the multiplexed buffer 52, the codedpicture buffer 53, the video decoder 54, the decoded picture buffer 55,and the secondary video plane 56.

<Structure of Playback Device 300: 2. Additional Hardware Component toController 45>

The controller 45 additionally includes a sync-setting unit 50.

The sync-setting unit 50 judges whether the sync_start_PTS_of_PlayItemincluded in the SubPlayItem information is set to an undefined value. Ifthe sync_start_PTS_of_PlayItem is set to an undefined value, theplayback device is brought into readiness to receive a rock operationmade by a user during the time a playback section of the Main Clipspecified by the PlayItem information is solely played back. The rockoperation is received via a remote controller and determines the startpoint of synchronous playback. Upon receipt of the rock operation, theundefined value of the sync_start_PTS_of_PlayItem is overwritten withthe value identifying the time at which the rock operation is received.In the case where the rock operation is defined as a selection of any ofbuttons presented on the playback image of the Main Clip, the time atwhich the operation of selecting the button is made is regarded as thetime of the rock operation.

Once the value of the sync_start_PTS_of_PlayItem is set in the abovemanner, the PL playback controller 48 can control playback of thePlayItem specifying the primary video and the SubPlayItem specifying thesecondary video. As a result, dynamically synchronized PiP playback isexecuted.

It should be naturally appreciated that if the value of thesync_start_PTS_of_PlayItem is statically set to a specific value, the PLplayback controller 48 can control playback of the PlayItem and theSubPlayItem so as to execute statically synchronized PiP playback.

<Structure of Playback Device: 2. Software Implementation>

This concludes the description of the components additionally providedin the second embodiment. Next, a description is given of improvementsmade in the software implementation of the second embodiment.

FIG. 45 is a flowchart illustrating processing steps for executing jumpplayback starting from an arbitrary point on the PlayItem timeline. Theflowchart in FIG. 45 is based on the flowchart illustrated in FIG. 30and is identical thereto except that the steps S4-S8 are replaced withsteps S31-S35. In the step S31, it is judged whether the EP_stream_typein the EP_map associated with the SubClip indicates the video type. Ifthe video type: 1 is indicated (step S31: YES), an offset α indicating apoint on the PlayItem timeline is converted into an offset β indicatinga corresponding point on the SubPlayItem timeline (=offsetα−sync_start_PTS_of_PlayItem) (step S32). Then, the offset β isconverted into a coordinate of a corresponding point on the SubCliptimeline (SubPlayItem_In_time+offset β) (step S33). Then, with referenceto the EP_map, the SubPlayItem_In_time+offset β is further convertedinto an address β of a corresponding GOP in the SubClip, which containsthe secondary video (step S34). Then, the playback device simultaneouslyaccesses the BD-ROM at the address a and the local storage 200 at theaddress β to read GOPs of the primary video and GOPs of the secondaryvideo, respectively (step S35).

In the step S31, if it is judged that the EP_stream_type does notindicate the video type: 1, the steps S4-S8 illustrated in FIG. 30 areperformed.

FIG. 46 schematically illustrates, in the same manner as FIG. 31, randomaccess to the Main Clip and to the SubClip. The SubClip containing thesecondary video has entry points at time intervals of less than 1 sec.Random access to the SubClip is made using those entry points. Sincerandom access to the SubClip can be made at about the same speed asrandom access to the Main Clip, jump playback of the SubClip is executedin synchronism with jump playback of the Main Clip.

As described above, during PiP playback in which is HD images of theprimary video are presented in combination with SD images of thesecondary video, a random access is made at a relatively high speed. Byvirtue of this high-speed random access capability, trick play can beexecuted even during PiP playback.

(Supplemental Notes)

Needless to say, the foregoing descriptions do not cover all the modesof practicing the present invention. The present invention can bepracticed also by any of the following modifications (A), (B), (C), (D),etc. It should be noted that the inventions recited in the claims of thepresent application are broadened or generalized descriptions of theabove-described embodiments and their modifications. The extent of thebroadening and generalization reflects the state of the art at the timeof filing the present application.

(A) Information processing performed by the functional components orshown in the flowcharts described in the above embodiments is actuallyrealized using hardware resources. Accordingly, the informationprocessing is said to be a highly advanced creation of technical ideasby which a low of nature is utilized, and thus satisfies therequirements of “program invention”.

Program Production According to the Present Invention

A program according to the present invention may be produced in thefollowing manner. First, a software developer writes, in a programminglanguage, a source program for implementing the flowcharts or thefunctional components described above. When writing the source programfor implementing the flowcharts or the functional components, thesoftware developer may use class structures, variables, array variables,and calls for external functions, in accordance with the syntax of thatprogramming language.

More specifically, for example, each loop shown in the flowchart may bedescribed using a FOR statement and each judgment may be described usingan IF statement or a SWITH statement, in accordance with the syntax. Thehardware control including the playback control of the decoder and theaccess control of the drive device can be described using CALLstatements for external functions provided by the hardware manufacturer.

The resulting source program is supplied as a file to a compiler. Thecompiler translates the source program into an object program.

The compilation involves processes of parsing, optimization, resourceallocation, and code generation. The parsing involves lexical analysis,syntactic analysis, and semantic analysis of the source program, andconversion of the source program into an intermediate program. Theoptimization involves processes of dividing the intermediate programinto basic blocks, and control flow analysis and data flow analysis ofthe intermediate program. The resource allocation involves a process ofallocating registers or memory of a target processor to variablesappearing in the intermediate program, whereby the intermediate programis adapted to the instructions sets of the processor. The codegeneration involves a process of converting intermediate statements ofthe intermediate program into program code, whereby an object program isgenerated.

The thus generated object program is composed of one or more lines ofcode for causing a computer to perform processing steps shown in theflowcharts referenced in the above embodiments or the processing stepsperformed by the functional components. The program code mentionedherein may be of any of a various types of codes including a native codeof the processor and Java (registered trademark) byte code. Theprocessing steps may be implemented by the program code in various ways.For example, when a step is implemented using an external function, aCALL statement for the external function comprises a line of the programcode. In addition, there may be a case where program code forimplementing one step is contained in two or more separate objectprograms. When a RISC processor, which is designed for handling reducedinstruction sets, is employed, each processing step of the flowchartsmay be implemented using a combination of an arithmetic instruction, alogical instruction, and a branch instruction, for example.

Once the object program is generated, the programmer activates a linker.The linker allocates memory areas for the object program and relatedlibrary programs, and binds them together to generate a load module. Thethus generated load module is to be read by a computer thereby to causethe computer to perform the processing steps shown in the aboveflowcharts or the processing steps performed by the functionalcomponents. Through the above processes, a program embodying the presentinvention is created.

(B) The program according to the present invention may be used in thefollowing manners.

(i) As Embedded Program

When the program according to the present invention is used as anembedded program, a load module that is equivalent to the program isrecorded into an instruction ROM, along with a basic input outputprogram (BIOS) and various pieces of middleware (operation systems). Theinstruction ROM is then built into the control unit and executed by theCPU. As a result, the program according to the present invention is usedas the control program of the playback device.

(ii) As Application Program

In the case where a playback device is provided with an internal harddisk, a Basic Input/Output System (BIOS) is already embedded within theinstruction ROM and middleware (operation system) is preinstalled ontothe hard disk. In addition, the playback device is provided with a bootROM for activating the system from the hard disk.

Thus, in this case, the load module is supplied to the playback devicevia a portable recording medium or a network, and the load module isinstalled as a single application program onto the hard disk. Then, theplayback device bootstraps to activate the operation system andinstructs the CPU to execute the application. In this way, the programaccording to the present invention is used as a single applicationprogram.

As described above, with a playback device having an internal hard disk,the program according to the present invention can be used as a singleapplication program. Thus, the program according to the presentinvention may be singly transferred, leased, or supplied over a network.

(C) Production and Usage of System LSI According to the PresentInvention

Generally, a system LSI is composed of a bare chip packaged on ahigh-density substrate. Alternatively, a system LSI may be composed of aplurality of bare chips that is packaged on a high-density substrate andhas an external structure just as a single LSI (this type system LSI isreferred to as a multichip module).

Focusing on the types of packaging, there are different types of systemLSIs called QFP (quad flat package) and PGA (Pin Grid Array). QFP is atype of system LSI with pins extending from all four sides of thepackage. PGA is a type of system LSI package with an array of pins thatare arranged on entire surface of the base of the package.

The pins act as an I/O interface with a drive device, a remotecontroller, and a television. The pins also act as an IEEE1394 interfaceand a PCI bus interface. Since the pins of the system LSI act asinterface, by connecting circuitry of a drive device or a playbackdevice, the system LSI plays a roll as the core of the playback device.

A bare chip packaged into a system LSI may be an instruction ROM, CPU, adecoder LSI implementing the functions of the components shown in thefigures of the internal structures according to the above embodiments.

As mentioned in the description given in relation to the use “AsEmbedded Program”, the instruction ROM stores a load module equivalentto the program of the present invention, a BIOS, and various pieces ofmiddleware (operation systems). The part to which the above embodimentsare particularly relevant is the load module equivalent to the program.Thus, by packaging, as a bare chip, the instruction ROM storing the loadmodule equivalent to the program, a system LSI according to the presentinvention is produced.

The details of the procedure for producing such system LSIs are asfollows. The first step is to make a circuit diagram of a part to beincorporated into a system LSI, based on the figures showing theinternal structures according to the above embodiments. The next step isto embody each component shown in the figures using circuit components,ICs, and LSIs.

The next step following the embodying each component is to design a busconnecting circuit elements, ICs, and LSIs, the peripheral circuitry,and interface with external devices. In addition, connecting lines,power lines, ground lines, and clock signal lines are designed. In thisprocess, operation timing of each component is adjusted in considerationof the LSI spec. In addition, some adjustment is made to ensure thebandwidth of each component. In this way, the circuit diagram iscompleted.

Once the circuit diagram is ready, the packaging design needs to be madenext. The packaging design is a process of designing a layout on asubstrate. To this end, the physical arrangement of the elements(circuit elements, IC, and LSI) shown in the circuit diagram isdetermined and the wiring on the substrate is also determined.

The packaging design mentioned herein may be done through the processescalled auto layout and auto wiring.

When a CAD apparatus is employed, the auto layout is carried out using adedicated algorithm called “centroid method”. In the process of autowiring, the connecting lines between the elements and pins shown on thecircuit diagram are implemented using metal foil and vias. With a CADapparatus, the wiring is completed using any of dedicated algorithmscalled “maze routing algorithm” and “line-search algorithm”.

After the packaging design is completed and the layout on the substrateis determined, the related data is converted into CAM data and suppliedto appropriate devices such as an NC machine tool. The NC machine toolincorporates the elements using System on Chip (SoC) or System inPackage (SiP) implementations. According to the SoC implementation,multiple circuits are baked on a single chip. According to the SiPimplementation, multiple chips are joined into a single package with,for example, resin. Through the above processes, a system LSI accordingto the present invention can be produced based on the figures showingthe internal structure of the playback device referenced in the aboveembodiments.

Note that integrated circuits produced in the above manner may bereferred to as IC, LSI, super LSI, or ultra LSI, depending on thepackaging density.

In addition, all or some of the components of each playback device ofthe present invention may be incorporated into a single chip. Inaddition, instead of the SoC and SiP implementations, the elements maybe integrated using a dedicated circuit or a general-purpose processor.For example, it is applicable to use an FPGA (Field Programmable GateArray) that can be programmed after an LSI is produced. It is alsoapplicable to use a reconfigurable processor that allows reconstructionof connection between circuit cells within the LSI and their settings.When any new circuit integration technology becomes available or derivedas the semiconductor technology advances, such new technology may beemployed to integrate the functional blocks of the present invention.One possible candidate of such new technology may be achieved byadapting biotechnology.

(D) According to the above embodiments, the recording medium accordingto the present invention is described as a hard disk. Yet, it should benoted that the features of the recording medium according to the presentinvention lie in the EP_map and the EP_stream map recorded thereon andthese features do not depend on the physical property of the hard disk.Any other recording medium is applicable as long as the recording mediumcan store the EP_map and the EP_stream_type and can be used incombination with a BD-ROM. Specific examples of such recording mediainclude: semiconductor memory cards, such as a CompactFlash card(registered trademark), a SmartMedia card, a Memory Stick card, aMultiMediaCard, and a PCM-CIA card. Alternatively, the recording mediumaccording to the present invention maybe (i) a magnetic disk, such as aflexible disk, SuperDisk, Zip, or Clik! or (ii) a removable hard diskdrive, such as ORB, Jaz, SparQ, SyJet, EZFley, or Microdrive.

(E) According to the above embodiments, each digital stream is an AVclip compliant with the BD-ROM standard. However, the digital stream maybe a VOB (Video Object) compliant with the DVD-Video standard or theDVD-Video Recording standard. A VOB is a program stream compliant withthe ISO/IEC 13818-1 standard, and is obtained by multiplexing a videostream and an audio stream. Alternatively, the video stream in an AVclip may be compliant with MPEG4 or WMV. Furthermore, the audio streamin an AV clip may be compliant with Dolby-AC3, MP3, MPEG-AAC, or dts.

(F) According to the above embodiments, the video stream is an MPEG4-AVCstream (which may also be referred to as “H.264” or “JVT”). Yet, thevideo stream may be of MPEG2 format or any other format such as the VC-1format as long as the video stream is solely decodable.

(G) In the case where the value of sync_start_PTS_of_PlayItem is set bythe sync-setting unit 50, it is desirable that the sync-setting unit 50perform the playback control as illustrated in FIGS. 47A-47C.

FIG. 47A illustrates the playback control for executing dynamicallysynchronized PiP playback. An arbitrary point on the PlayItem timelineis determined as the start point for synchronization with theSubPlayItem. When the current playback point reaches the time indicatedby the sync_start_PTS_of_PlayItem, the playback device starts decodingthe secondary video and overlays playback of the secondary video withplayback of the primary video.

FIG. 47B illustrates dynamically synchronized PiP playback executed inthe case where the rock point is once passed and reached again in thenormal playback mode as a result of rewinding. In other words, thefigure illustrates how dynamic synchronization in PiP playback isestablished in the case where the current playback point once passes therock point in the normal playback mode and then is rewound by the userbeyond the rock point, and reaches the rock point for the second time inthe normal playback mode.

When the rock point is reached for the first time, PiP playback isstarted. PiP playback is stopped at the moment when the rewindingoperation is made. Then, playback is resumed in the normal playback modeand PiP playback is not executed even when the rock point is reached forthe second time.

FIG. 47C illustrates PiP playback executed in the case where theplayback section of the secondary video is located at some pointsubsequent to the end point of the playback section of the primaryvideo. In this case, the last picture of the primary video iscontinually presented until playback of the secondary video ends.Alternatively, it is applicable to end playback of the secondary videoat the time when playback of the primary video ends.

(H) In the case where the current playback point of the secondary videois not changed at relatively short time intervals, the PlayListinformation needs to define many pieces of timing information andposition information. In this case, it is preferable that the PlayListinformation additionally includes User Private information and PLMarkinformation compliant with the BD-RE specifications and defines thePiP_position and PiP_size in the additionally provided information.Then, the scaler 57 and the compositor 58 can perform the resizingprocess and the composition process according to the user privateinformation and the PLMark information.

(I) It is applicable to present the primary video and the secondaryvideo on two separate screens rather than one and the same screen. Asdescribed above, the main video is a portion of the Main Clip specifiedby the MainPath information in the PlayList information, whereas thesecondary video is a portion of the SubClip specified by the SubPlayItemalso in the PlayList information. In addition, it is applicable topresent the primary video with SD images and the secondary video withthe HD images.

(J) According to the second embodiment, the Main Clip containing theprimary video is supplied in from of BD-ROM, and the SubClip containingthe secondary video is supplied to be stored on the local storage 200.Yet, it is applicable that the SubClip is sorted on the BD-ROM andsupplied to the playback device together with the Main Clip.

In addition, it is applicable that the secondary video stream ismultiplexed with the primary video stream to constitute one AV clip.

INDUSTRIAL APPLICABILITY

The recording medium and the playback device according to the presentinvention may be employed for personal use in a home theater system, forexample. Yet, the present invention may be manufactured in volume inaccordance with the internal structures disclosed in the aboveembodiments. Thus, the recording medium and the playback device of thepresent invention can by industrially manufactured or used on anindustrial scale. In view of the above, the recording media and theplayback devices of the present invention have industrial applicability.

1. A playback device for executing trick play of a main stream and asubstream, a portion of the main stream being defined as a primaryplayback section, and a portion of the substream being defined as asecondary playback section, the primary and secondary playback sectionsbeing defined in playlist information, the playlist informationincluding synchronous information, the synchronous information includingtiming information that indicates a synchronization point on a timelineof the primary playback section, the synchronization point being a pointfor starting synchronous playback of the primary playback section withthe secondary playback section, the substream being associated with anentry map, and a start point of the trick play being defined on thetimeline of the primary stream, the playback device comprising: a firstconversion unit operable to convert the start point into a correspondingaddress on the main stream; a second conversion unit operable to convertthe start point into a corresponding point defined on a timeline of thesecondary playback section and to further convert the correspondingpoint into a corresponding address on the substream, the conversion intothe corresponding point being performed by using the synchronousinformation included in the playlist information, and the conversioninto the corresponding address being performed by using the entry mapassociated with the substream; a reading unit operable to read the mainstream and substream starting from the respective addresses obtained bythe first and second conversion units; and a playback unit operable toplay back the main stream and the substream read by the reading unit,wherein the main stream is a digital stream carrying a first video, thesubstream is a digital stream carrying a second video, the playback unitincludes: a first decoder operable to decode the main stream to obtainthe first video; and a second decoder operable to decode the substreamto obtain the second video; and the playback device further comprises: acomposite unit operable to execute Picture in Picture playback byoverlaying, on a same screen, playback of the first video obtained bythe first decoder with playback of the second video obtained by thesecond decoder.
 2. A program for causing a computer to execute trickplay of a main stream and a substream, a portion of the main streambeing defined as a primary playback section, and a portion of thesubstream being defined as a secondary playback section, the primary andsecondary playback sections being defined in playlist information, theplaylist information including synchronous information, the synchronousinformation including timing information that indicates asynchronization point on a timeline of the primary playback section, thesynchronization point being a point for starting synchronous playback ofthe primary playback section with the secondary playback section, thesubstream being associated with an entry map, and a start point of thetrick play being defined on the timeline of the primary stream, theprogram comprising code operable to cause the computer to perform: afirst conversion step of converting the start point into a correspondingaddress on the main stream; a second conversion step of converting thestart point into a corresponding point defined on a timeline of thesecondary playback section and further converting the correspondingpoint into a corresponding address on the substream, the conversion intothe corresponding point being performed by using the synchronousinformation included in the playlist information, and the conversioninto the corresponding address being performed by using the entry mapassociated with the substream; a reading step of reading the main streamand substream starting from the respective addresses obtained in thefirst and second conversion steps; and a playback step of playing backthe main stream and the substream read in the reading step, wherein themain stream is a digital stream carrying a first video, the substream isa digital stream carrying a second video, and the playback stepincludes: a first decoding step of decoding the main stream to obtainthe first video; and a second decoding step of decoding the substream toobtain the second video, and the program further comprises code operableto cause the computer to perform: a composite step of executing Picturein Picture playback by overlaying, on a same screen, playback of thefirst video obtained in the first decoding step with playback of thesecond video obtained in the second decoding step.
 3. A playback methodfor executing trick play of a main stream and a substream, a portion ofthe main stream being defined as a primary playback section, and aportion of the substream being defined as a secondary playback section,the primary and secondary playback sections being defined in playlistinformation, the playlist information including synchronous information,the synchronous information including timing information that indicatesa synchronization point on a timeline of the primary playback section,the synchronization point being a point for starting synchronousplayback of the primary playback section with the secondary playbacksection, the substream being associated with an entry map, and a startpoint of the trick play being defined on the timeline of the primarystream, the playback method comprising: a first conversion step ofconverting the start point into a corresponding address on the mainstream; a second conversion step of converting the start point into acorresponding point defined on a timeline of the secondary playbacksection and further converting the corresponding point into acorresponding address on the substream, the conversion into thecorresponding point being performed by using the synchronous informationincluded in the playlist information, and the conversion into thecorresponding address being performed by using the entry map associatedwith the substream; a reading step of reading the main stream andsubstream starting from the respective addresses obtained in the firstand second conversion steps; and a playback step of playing back themain stream and the substream read in the reading step, wherein the mainstream is a digital stream carrying a first video, the substream is adigital stream carrying a second video, and the playback step includes:a first decoding step of decoding the main stream to obtain the firstvideo; and a second decoding step of decoding the substream to obtainthe second video, and the playback method further comprises: a compositestep of executing Picture in Picture playback by overlaying, on a samescreen, playback of the first video obtained in the first decoding stepwith playback of the second video obtained in the second decoding step.4. A method for recording to a recording medium, comprising: a step ofgenerating application data; and a step of recording the applicationdata onto a recording medium, wherein the application data includesplaylist information, a plurality of digital streams, and an entry map,the playlist information defines a playback section of each of aplurality of digital streams and includes main-path information andsub-path information, the main-path information designates one of thedigital streams as a main stream and defines a portion of the mainstream as a primary playback section, the sub-path informationdesignates another one of the digital streams as a substream and definesa portion of the substream as a secondary playback section that is to besynchronously played back with the primary playback section, on therecording medium, the entry map is associated with the one of theplurality of digital streams designated as the substream, the entry mapindicates a plurality of entry times on a timeline of the substream inone-to-one correspondence with a plurality of entry points on thesubstream, the sub-path information includes synchronous information,the synchronous information includes timing information that indicates asynchronization point on a timeline of the primary playback section, thesynchronization point being a point for starting synchronous playback ofthe primary playback section with the secondary playback section, themain stream is a digital stream carrying a first video, the substream isa digital stream carrying a second video, and when reading the recordingmedium, the playback device executes Picture in Picture playbackstarting from the synchronization point in accordance with the playlistinformation, by overlaying playback of the first video with playback ofthe second video on a same screen.